SELECTION OF INDIVIDUALS FROM A POOL OF 
CANDIDATES IN A COMPETITION SYSTEM 



FIELD OF THE INVENTION 



[0001] The present invention relates to the evaluation of human resources, and 
more particularly to the automation-enhanced selection of individuals from a pool of 
competition candidates based on evaluations of cognitive and non-cognitive 
variables relative to a competition standard. 



BACKGROUND OF THE INVENTION 



[0002] The appropriate selection of individuals or candidates for employment, 
appointment to political positions, promotions, receiving awards or scholarships, and 
so on has traditionally been a difficult problem. 

[0003] One trend facing competition systems is that of ever-increasing numbers 
of applications being received by competition systems. This results in problems of 
data handling, of selecting the best candidates from an ever-increasing number of 
highly-qualified applicants, and also of eliminating an ever-increasing number of 
lower-quality, ineligible, or incompletely-responsive candidates. 
[0004] The problem of candidate selection is compounded by candidate fraud 
which is becoming increasingly sophisticated. The forging of documents, the use of 
documents or credentials of others having similar names or demographic 
information, and so on complicates competition systems due to the detection and 
resolution requirements which must be imposed, as well as the resultant unfairness 
when such fraud goes undetected. 

[0005] Another problem is the occurrence of duplicate forms. This often occurs 
as individuals apply multiple times in order to correct information or perhaps even in 
the belief that this can aid their chances of success. 

[0006] The detection of both duplicates and fraud is further complicated, 
however, in that situations arise where two or more legitimate candidates sometimes 
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appear identical from their submitted papers. Such situations often arise in 
connection with twins having similar or identical names, for example. 
[0007] The selection of candidates for positions of future responsibility such as 
political appointees, military or civilian officers or leaders, corporate management, 
5 academic scholars, and so on has at its root the problem of selecting the best 
candidate based on predicted future performance. Studies have shown that non- 
traditional variables such as non-cognitive variables are sometimes a better predictor 
of future performance than traditionally-used variables. 

[0008] The use of these variables, as well as the use of traditional but subjective 
10 variables, such as essay tests, has a problem because the nature of their evaluation is 
subjective and thus difficult, even for a single evaluator, to do in a fair and 
consistent manner. This problem increases with the number of evaluators and 
encompasses the problems of evaluator selection, training, and monitoring. Studies 
have shown, for example, that evaluators sometimes vary their decision 
1 5 methodologies in their evaluations, especially just before breaks or at the end of the 
day. Furthermore, studies have also shown that evaluators may evaluate non- 
cognitive variables more harshly for candidates of one ethnic or cultural background 
as compared with candidates of other ethnic or cultural backgrounds. The 
monitoring of evaluators is additionally complicated as system generally don't have 
20 the infrastructure necessary to allow real-time monitoring in a non-invasive way or 
the ability to implement corrective measures in real-time. 

[0009] Additionally, traditional systems are lacking in methodologies to track 
both competition winners and competition losers for later use in producing statistical 
support for and improvement of system selection criteria. 
25 [0010] From the above, it is evident that improvements are needed in the areas 
of data management (i.e. duplicate identification, false-duplicate identification, and 
fraud identification), candidate pool reduction, evaluator management, candidate 
selection, and winning-candidate progress tracking. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



FIG. 1 shows a block diagram of an embodiment of a system for the selection of 

individuals from a pool of candidates; 
FIG. la shows a block diagram of an embodiment for database 104 of FIG. 1; 
5 FIG. 2 shows a block diagram of an embodiment for a high-level software 

architecture for the system of FIG. 1; 
FIGS. 3a-3b show a block diagram in greater detail of the software architecture of 
FIG. 2; 

FIGS. 3c-3d show the block diagram of FIGS. 3a-3b emphasizing the 
10 interconnections of the reporting module (280). 

FIG. 4 shows a general flowchart of steps traversed by an exemplary method in 

operation of the systems of FIGS. 1-2; 
FIG. 5 shows a flowchart of steps traversed by an exemplary method in the operation 

of electronic reception of documents in the systems of FIGS. 1-2; 
1 5 FIG. 6 shows a flowchart of steps traversed by an exemplary method in the operation 

of hardcopy reception in the systems of FIGS. 1-2; 
FIG. 7 shows a flowchart of the steps traversed in an exemplary algorithm for the 

storing and duplicate detection for the submission procedures shown in 

FIGS. 5-6; 

20 FIG. 7a shows a flowchart of an exemplary embodiment for possible duplicate 
document review; 

FIG. 8 shows a flowchart of steps traversed in an exemplary algorithm the grouping 
and level 1 filtering for candidate package completeness shown in FIG. 4; 

FIG. 9 shows a flowchart of steps traversed in an exemplary algorithm in the 
25 operation of the level 2-3 filtering of electronic documents shown in FIG. 4; 

FIG. 1 0 shows a flowchart of steps traversed in an exemplary method in operation of 
the reading process of FIG. 4; 

FIG. 1 1 shows a flowchart of steps traversed in an exemplary method in operation of 
the selection and training of readers to participate in the process of FIG. 8; 
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FIGS. 12a- 12b show a flowchart of steps traversed in an exemplary algorithm in 
operation of finalist selection of FIG. 4; 

FIGS. 12c-12e shows an example of the steps traversed in an application of the tie- 
breaking procedure of FIG. 12b. 
5 FIG. 13 shows a flowchart of steps traversed in an exemplary method in operation of 
competition winner confirmation of FIG. 4; 

FIG. 14 shows a flowchart of steps traversed in an exemplary method in operation of 
competition winner tracking of FIG. 4; 

FIG. 14a shows a flowchart of the steps traversed in an exemplary method for the 
1 0 tracking of competition winners and non-selected applicants; 

FIG. 1 5 shows an exemplary display presented to an evaluator; 

FIGS. 16-17 show exemplary displays for use during determination of evaluator 

eligibility; 

FIGS. 18-19 show exemplary displays for use during determination of candidate 
1 5 eligibility; 

FIGS. 20-21 show exemplary displays for use during evaluation of candidate 
packages; 

FIGS. 22-25 show exemplary displays of candidate progress detail reports; 
FIGS. 26-28 show exemplary displays of completed applicant package reports; 
20 FIGS. 29-30 show exemplary displays of possible duplicate document reports; 
FIGS. 3 1-34 show exemplary display of candidate detail listing reports; 
FIG. 35 shows an exemplary display of a nominator floater with possible key form 
report; 

FIGS. 36-37 show exemplary display of nominator floater detail reports; 
25 FIGS. 38-40 show exemplary display of nominator detail reports; 

FIG. 41 shows an exemplary display of a recommender floater with possible key 
form report; 

FIGS. 42-43 show exemplary display of recommender floater detail reports; and 
FIGS. 44-46 show exemplary display of recommender detail reports. 
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DETAILED DESCRIPTION 



[0011] Shown and described are embodiments, automated and partially- 
automated, of a system and method for selecting one or more candidates from a pool 
of candidates. It is noted that while this disclosure illustrates the present invention 
5 by examples drawn mainly from academic scholarship competitions, the present 
invention has application in many fields outside of academics such as, but not 
limited to: employee hiring; employee to job, position, or task matching; military 
officer selection; political appointee selection; personal date selection; personal 
friend selection; and so forth. 

1 0 [0012] Referring to FIG. 1 , shown is a block diagram of an embodiment for a 
competition system (100) for the selection of individuals from a pool of candidates. 
[0013] Shown are system processing hardware (1 02), database hardware (1 04), a 
non-select segment (1 12), a candidate segment (1 14), a scholar segment (1 16), a 
client device (120), a network (122), network interface hardware (124), an envelope 

15 (130) representing postal delivery, a scanning and processing system (132), a 

workstation (140), a workstation (142) labeled "workstation n", a correspondence 
printer (150), and a check printer (1 52). 

[0014] The system processing hardware (102) is coupled to the database 
hardware (104). The database hardware (104) as shown comprises the non-select 

20 segment (1 12), the candidate segment (1 1 4), and the scholar segment (116). The 

client device (120) is coupled to the network (122). The network (122) is coupled to 
the network interface hardware (124), which, in turn, is coupled to the system 
processing hardware (102). The envelope (130) representing postal delivery is 
coupled to the scanning system (132) which, in turn, is coupled to the system 

25 processing hardware (102). The workstation (140) and the workstation (142) labeled 
"workstation n" are both coupled to the system processing hardware (102). The 
correspondence printer (150) and the check printer (152) are also both coupled to the 
system processing hardware (102). 

[0015] The system processing hardware (102) carries out processing required for 
30 the competition system (100). The system processing hardware (102) can be 
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implemented by any hardware suitable for running general purpose software such as, 
but not limited to, any commercial computing hardware. By way of example, most 
computing systems having one or more Intel-, AMD-, or Motorola-based processors 
and targeted for general home or business use can be used for the system processing 
5 hardware (102). 

[0016] The database hardware (104) stores information relating to candidates 
required in carrying out the processing of competitions. The exact nature of the 
storage architecture is not dependant on the present invention and can take any form 
necessary to the particular implementation. In one embodiment, as shown, the 

10 database hardware (104) is segmented to manage individuals during various stages 
of consideration in the competition system (100). In this embodiment, data is 
marked to segment or group the data. The data is segmented or grouped into the 
categories which include applicants (the applicant category or segment (1 14)), 
applicants who are not selected (the non-select category or segment (1 12)), and 

1 5 candidates selected as competition winners (the competition- winner category or 

segment (116) also called the scholar category or segment in one embodiment). In 
one variation, the database can be any suitable commercial database product such as, 
but not limited to, products by Oracle, Sybase, or Microsoft. 
[0017] In another embodiment, the database hardware (1 04) is physically 

20 differentiated into two or more separate databases as required. In one embodiment, 
the database hardware (104) includes a staging or applicant database, a non-select 
database, a candidate or nominee database (for applicants who have passed 
qualification and eligibility processes), and a competition-winner or scholar 
database. This embodiment is used, for example, where existing hardware capacities 

25 mandate such an approach or in systems which have an absolute requirement of data 
separation such as for security reasons. In systems requiring higher separation of 
data, a fifth database for the storage of finalists prior to confirmation, a finalist 
database, can be added. 

[0018] In an embodiment, applications, to be complete, comprise two or more 
30 documents. In a further variation, an applicant package (also called a candidate 
package or profile) includes an applicant form (submitted by the applicant), also 
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called a nominee or candidate form, a nominator form (submitted by an individual 
with academic knowledge of the applicant such as a school teacher or administrator), 
and a recommender form (submitted by an individual with personal knowledge of 
the applicant such as a family friend. It is noted that school officials and teachers 
5 can also be nominators). In this embodiment, the applicant segment (1 14) comprises 
applicant-related submissions received from both the electronic and hardcopy 
submissions. The non-select segment (112) comprises applicant packages for 
applicants who ultimately do not win a competition, i.e., for applicants who are 
ultimately not selected or for applicants having incomplete or ineligible applicant 
1 0 packages. The competition- winner segment (116) (also called the scholar segment 
in academic scholarship embodiments) comprises candidate packages relating to 
those candidates who ultimately win a competition. 

[0019] When application documents are received by the system, they are stored 
into the database (104), associated with any other records (the database versions of 

1 5 submitted documents) relating to the same applicant, and marked as being in the 

applicant segment (114). Later, the associated records (i.e. the applicant packages) 
are submitted to qualification and eligibility procedures. Applicant packages do not 
pass either the qualification and eligibility procedures are marked as being in the 
non-select segment (112). Later still, the applicant packages which have not been 

20 non-selected are evaluated, scored, and one or more competition winners selected. 
In one embodiment, once an applicant is selected as a competition finalist, he or she 
is referred to as a candidate and their applicant package is referred to as a candidate 
package. The candidate packages for those candidates who ultimately win a 
competition are marked as being in the competition- winner segment (116) upon 

25 certification (verification or confirmation of data supplied in the applicant, 

nominator, and recommender forms) and acceptance of competition winning status 
(i.e. acceptance of the competition award) by the candidate. Further to this 
embodiment, candidates who decline the competition award, or who are not 
certified, are designated as non-selects and their candidate packages are marked as 

30 being in the non-select segment (112). 
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[0020] In one embodiment, in order to keep track of the status of individuals 
during the competition process, various lists are maintained or the ability to create 
them at any time is preserved. These lists include an applicant list for listing all 
applicants for which any document or form has been submitted, a non-select list for 
5 listing all individuals who ultimately are not winners of the competition, a candidate 
list for all applicants passing an initial qualification and eligibility process, a finalist 
list for listing all candidates selected as finalists, and a competition-winner list for 
listing all finalists who pass a confirmation and acceptance process. Further to this 
embodiment, these lists are protected and each require (possibly different) special 

10 system privileges to have access. In a variation of this embodiment, each list can 

include various data regarding the individuals listed. The non-select list can include, 
for example, the date and stage within the competition at which the individual was 
non-selected or the competition-winner list can include dates at which the 
competition winner is expected to submit requalification documents. 

1 5 [0021] Information relating to applicants can be submitted by any conventional 
method such as by postal mail or electronic submission. Applicant information 
submitted by postal mail generally takes the form of a paper document (including 
predefined forms after user completion or nonstandard documents such as letters), 
but other manifestations, such as, but not limited to, computer readable media 

20 (containing wordprocessing files or even image scans of paper documents) are 
possible. As used herein, the term "document" refers to any submissions of 
applicant information and includes, for example, paper submissions, scanned images 
on computer-readable media, or any other media or vehicle containing information 
to be incorporated into the competition system. 

25 [0022] Electronic submission begins when a submitted document is transmitted 
by the client device (120) via the network (122) to the competition system (1 00) 
where it is received via the network interface hardware (124). The client device 
(120) can be any user device able to initiate document submissions. By way of 
example, a client device (120) can be, but is not limited to, a computer, a handheld 

30 device such as a personal digital assistant (PDA), or a computer network such as a 
business, college, or university intranet. The network (122) can be any electronic or 
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computer network or interconnection of networks such as one or more intranets or 
internets. In one embodiment, the network (122) is the Internet. The network 
interface hardware (124) is the physical connection of the competition system (100) 
to the network (122). The network interface hardware (124) can be any device such 
as, but not limited to, a modem, a Tl interface, and so forth. Electronic submissions 
are discussed in greater detail later herein in reference to FIG. 5. 
[0023] Documents submitted by postal mail are scanned in at the scanning 
system (132) and thereby incorporated into the system. In the case of submissions 
on computer-readable media, the candidate information is read in at a computer 
workstation such as the workstation (140) or (142). Hardcopy submissions are 
discussed in greater detail later herein in reference to FIG. 6. 
[0024] In one embodiment, the overall selection process has several stages 
including an application stage, a qualification stage, an evaluation stage, a ranking 
and selection stage, a finalist stage, a confirmation stage, and a tracking or 
monitoring stage. The application stage is where individual applications are 
received by the competition system (100) as discussed previously. The qualification 
stage implements basic qualification and eligibility requirements and is discussed in 
greater detail later herein in reference to FIGS. 8 and 9. The evaluation stage 
implements evaluation of qualified applicants and is discussed in greater detail later 
herein in reference to FIG. 10. The ranking and selection stage implements the 
ranking of evaluated candidates and the selection of finalists. Finalists are 
candidates who are selected after the evaluation process, but who must pass a 
certification process before becoming unqualified competition winners. The ranking 
and selection process is discussed in greater detail later herein in reference to FIGS. 
12a- 12b. 

[0025] Once finalists have been identified, the confirmation stage implements 
confirmation of the qualifications of finalists and allows for ultimate acceptance by 
confirmed finalists. The confirmation of finalists is discussed in greater detail later 
herein in reference to FIG. 13. The tracking or monitoring stage implements the 
tracking of both competition winners (i.e., monitoring the progress of finalists who 
have accepted their competition awards) and non-selected candidates (for 
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comparison purposes). Competition award payment and correspondence, such as 
notice letters and requests for progress reports, are generated by the payment and 
correspondence generation system during the tracking and monitoring stage. The 
tracking of competition winners is discussed in greater detail later herein in 
5 reference to FIG. 14. 

[0026] The workstation (140) facilitates user use of the competition system 
(1 00). The competition system can be used though a single workstation but can also 
be implemented with any number of workstations necessary as indicated by the 
workstation (142) labeled "workstation n". The workstation (140) can be any 

1 0 computing device able to allow display and entry of information such as, but not 

limited to, a personal computer, a handheld computing device, a laptop computer, a 
notebook computer, a SPARC (tm) station, etc. The workstation ( 1 40) has at least a 
display, such as a text or graphical monitor to display information and one or more 
input devices such as a keyboard and a mouse to allow entry and manipulation of 

1 5 information. 

[0027] The correspondence printer (150) prints correspondence and the check 
printer (152) prints checks. While two printers are shown, any suitable number of 
printers can be used such as a report printer for use by the reporting module (280) 
discussed in greater detail later herein in reference to FIG. 2. Alternatively, printing 
20 can be partially or totally outsourced. 

[0028] Referring to FIG. la, shown is a block diagram of an embodiment for 
database (104) of FIG. 1. 

[0029] Shown are the database (104), an applicant database (162), a scholar 
database (164), an analytical database (166), an assessment database (168), a 

25 research database (170), and an archive database (172). 

[0030] In this embodiment, the applicant database (1 62) and the scholar database 
(164) are equivalent, respectively, to the applicant segment (114) and the 
competition-winner segment (1 16) of FIG. 1 . The analytical database (166) stores 
information relating to analytical studies of candidates in the system to determine 

30 trends. The assessment database (168) stores information relating to assessments 
and other input from vendor, donor, and other entities regarding the competition 
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system (1 00). The research database (170) stores information relating to research 
regarding improvement of the rubric or application thereof used by the competition 
system (100). Generally, the results stored in the analytical database (166) is 
developed from the data stored in the research database (1 70). The archive database 
5 (1 72) stores any data which needs to be archived. 

[0031] Referring to FIG. 2, shown is a block diagram of an embodiment of a 
high-level software architecture for the system of FIG. 1 . 

[0032] Shown are a document reception subsystem (210), a database interfacing 
subsystem (220), a qualification and eligibility subsystem (230), an evaluation and 
1 0 selection subsystem (240), a confirmation subsystem (250), a tracking and re- 
qualification subsystem (260), a correspondence and check printing interface 
subsystem (270), a reporting module (280), an external interface module (290), and a 
financial subsystem (295). 

[0033] The document reception subsystem (210) is coupled to the database 

1 5 interfacing subsystem (220), the confirmation subsystem (250), and the tracking and 
re-qualification subsystem (260). The qualification and eligibility subsystem (230) 
is coupled to both the evaluation and selection subsystem (240) and the database 
interfacing subsystem (220). The evaluation and selection subsystem (240) is also 
coupled to the confirmation subsystem (250) and the database interfacing subsystem 

20 (220). The confirmation subsystem (250) is additionally coupled to the database 
interfacing subsystem (220), the tracking and re-qualification subsystem (260), and 
the financial subsystem (295). The tracking and re-qualification subsystem (260) is 
also additionally coupled to the database interfacing subsystem (220). The financial 
subsystem (295) is coupled to the confirmation subsystem (250), the tracking and re- 

25 qualification subsystem (260), and the correspondence and check printing interface 
subsystem (270). The reporting module (280) and the external interface module 
(290) are coupled to each other and both are coupled to all the other subsystems. 
[0034] The document reception subsystem (210) handles the incorporation of 
incoming documents into the competition system (100). While documents can be 

30 received in a variety of ways, such as electronically or by mail, the document 

reception subsystem (210) incorporates the documents into the system in a flat file 
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format which is an electronic file containing the data of the original document. Flat 
files contain their data in a predefined format understandable by the various 
subsystems of the competition system (100). In one embodiment, the flat file format 
is a comma delimited format such as a comma separated value (CSV) format. This 
5 stored version of a document is referred to as a record, database record, or document 
record. 

[0035] The qualification and eligibility subsystem (230) implements basic 
qualification and eligibility requirements. The qualification and eligibility 
subsystem (230), in one embodiment, has one or more filters. A filter applies one or 

10 more criteria to an applicant package and determines whether the applicant package 
meets or does not meet the criteria. In one embodiment, applicant packages which 
do not meet any criteria are disqualified and marked so as to be in the non-select 
segment (112). In other embodiments, some criteria can be discretionary and 
applicants are given the chance to correct deficiencies. In one embodiment, the 

1 5 filters initially determine applicant package completeness (both data completeness 

and document record completeness). In this embodiment, once an applicant package 
is determined to have completeness, the applicant package is marked to be in the 
candidate segment (114) and is now called a candidate package. After this, further 
filters apply basic eligibility and discretionary eligibility requirements on the 

20 candidate packages. 

[0036] The evaluation and selection subsystem (240) implements evaluation and 
scoring of candidate packages. The evaluation and selection subsystem (240) further 
compare scored candidate packages against each other and determines competition 
finalists. Competition finalists are those candidates which have been selected as 

25 competition winners, but are still required to pass a confirmation and acceptance 

procedure. In another embodiment, the confirmation and acceptance procedures are 
dispensed with, and the candidates selected by the evaluation and selection 
subsystem (240) directly become competition winners. This embodiment is used in 
competitions which do not place further responsibility or requirements on 

30 competition winners. By way of example, awards of recognition, such as a literary 
award, can be given without requiring confirmation or acceptance. 
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[0037] The confirmation subsystem (250) implements procedures for confirming 
finalists as competition winners and also implements the printing of correspondence 
alerting a finalist of his or her status and notifying them of the requirements and 
deadlines for submitting confirmation documents. In one embodiment, if a finalist 
5 passes the confirmation and acceptance procedures, the finalist becomes a 

competition winner and the candidate package of the finalist is marked as being in 
the confirmation-winner segment. 

[0038] The tracking and re-qualification subsystem (260) implements the 
procedures for tracking competition winners which includes the printing of 

1 0 correspondence notifying and alerting a competition winner to deadlines and 

requirements for staying qualified and the actual re-qualification determination for 
competition winners. Re-qualification is imposed on competition winners for whom 
a continuing performance criteria is required. By way of example, an academic 
scholarship may require a continuing minimum level of performance as measured by 

1 5 grade point average (GPA) or a minimum level of course difficulty. 

[0039] The correspondence and check printing interface subsystem (270) 
interfaces between other subsystems and the printing hardware for printing 
correspondence, such as on the correspondence printer (150), and checks, such as on 
the check printer (152). 

20 [0040] The database interfacing subsystem (220) allows the other subsystems 
access to the database hardware (104) of the competition system (100). 
[0041] The reporting module (280) generates any reports documenting processes 
or results which take place. These reports include, but are not limited to, error 
reports such as scanning error or recognition error reports, status reports such as 

25 applicant status reports, audit reports (e.g. audit trails), possible duplicate reports, 

complete and incomplete candidate package reports, measurement reports, candidate 
detail reports, unmatched document (i.e. floater) reports, progress reports, 
documentation reports, and donor or other external reports. Documentation reports 
can include reports which document when and what procedures are used during 

30 competitions. Such documents can be valuable in a legal sense for sensitive 

competitions in which the losing candidates may attempt to make allegations of 
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impropriety. Donor and other external reports can be produced ad hoc to provide 
specific and detailed information to the recipients. Donors, for example, may wish 
to know how their donations are used and reports addressing this may include 
amounts paid out, when, and how apportioned. 
5 [0042] The external system interface module (290) enables the system to 

interface to other external application systems such as, but not limited to, financial 
systems, human resources (HR) systems, and registration systems. 
[0043] The financial subsystem (295) implements the financial and accounting 
aspects of the system including keeping track of donations, award payments, and 

10 system costs. In an embodiment for the payment of academic scholarships, the 
financial subsystem (295) manages the award payments to scholars. In this 
embodiment, once an applicant is confirmed as a finalist, the financial subsystem 
(295) begins keeping track of the scholar. The financial subsystem (295) does this in 
yearly installments by reviewing the scholars package (i.e. the set of documents 

1 5 retained by the system on the scholar) and calculates a current term award by 

summing the monetary amounts of all scholarships (including 3 rd party scholarships), 
grants, and so forth and subtracts this amount from the total tuition amount. The 
resulting amount is divided by the number of semesters, trimesters, etc. applicable to 
arrive at a base award amount, also called the current term award. Note that in this 

20 embodiment, money from programs such as loans, work study, etc. are not included 
in the scholarship and grant calculation as the goal of this embodiment is to allow 
the student to maximize study time without incurring debt for later repayment. After 
arriving at the current term award, the financial subsystem (295) calculates an 
adjustment to the current term award which takes into account the effects of short- 

25 term scholarships, differences in tuition due to special circumstances such as study 
abroad or changing institutions, etc. to arrive at a corrected current term award. This 
amount is then certified and results in payment generated and given to the scholar. 
[0044] The financial subsystem (295) updates the previous calculations during 
the academic year at each term by reviewing the revised scholar award package and 

30 calculating a supplemental award based on the review of the revised scholar award 
package. The revised scholar award package is generated by adding term-specific 
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information to the scholar award package. It is noted that the financial subsystem 
(295) accommodates scholarship payments for summer terms, study abroad. 
[0045] Referring to FIGS. 3a-3b, shown is a block diagram showing details of 
the subsystems of the software architecture of FIG. 2. 
5 [0046] Shown in FIG. 3a are a document reception subsystem (210), a 

qualification subsystem (230), a selection of competition winners subsystem (240), 
and a database interfacing subsystem (220). Each of these subsystems comprise 
various modules that are discussed individually as follows. 
[0047] The document reception subsystem (210) is shown having a network 
1 0 interface module (21 1), a manual input module (212) a scanner interface module 
(213), a field extraction and OCR module (214), and an image storage (215). OCR 
is an acronym for optical character recognition. 

[0048] The scanner interface module (213) is coupled to the field extraction and 
OCR module (214). The field extraction and OCR module (214) is coupled to the 

1 5 image storage (215). The network interface module (211), manual input module 
(212), and the field extraction and OCR module (214) are all coupled to the 
duplicate check / database interface module (221). The image storage (215) is 
coupled to both the confirmation module (252) in FIG. 3b (via the connector "A"), 
and the re-qualification module (263) in FIG. 3b (via the connector "B"). 

20 [0049] The document reception subsystem (2 1 0) handles the incorporation of 
incoming documents into the competition system. In one embodiment, as shown, 
documents can be incorporated by any of three methods: electronic submission, 
hardcopy submission, and manual entry. Electronic submissions are received 
through the network interface module (211) and can originate from any electronic 

25 source such as an internet (e.g. the Internet) or an intranet. Electronic submissions 

can also be accepted from computers coupled directly to the competition system (and 
locally transferring documents in flat file format received on magnetic disc or 
compact disc, for example) such as the workstation (140) or the workstation (142). 
In one embodiment, the process of electronic submission directly produces the flat 

30 file format of the inputted form as the user (i.e. the applicant or, in one embodiment, 
the recommender or nominator) interacts with the competition system (100) in 
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entering data. Electronic data entry by users is discussed in greater detail later herein 
in reference to FIG. 5. 

[0050] From the network interface module (211), flat files are transferred to the 
duplicate check / database interface module (221). Incorporation of flat files is 
5 discussed in greater detail later herein in reference to FIG. 7. 

[0051] Documents can also be submitted in paper or other hardcopy form 
including bit-image data on computer readable media such as on magnetic disc or 
optical disc, for example. Bit-image data refers to files not in flat format, such as 
bit-mapped images. Media containing bit-mapped image data generally results from 

10 external scanning of paper documents and storage onto computer-readable media, 
most likely by the candidate himself or herself. In the embodiment as shown, the 
process of hardcopy submission is received through the scanner interface module 
(213). Paper, or other hardcopy documents requiring scanning, are first scanned in 
at one or more scanner modules and incorporated onto the system via a scanner 

15 interface module (213). Pre-scanned forms submitted on computer-readable media 
are simply read into the system. Incorporation of documents via scanning is 
discussed in greater detail later herein in reference to FIG. 6. 
[0052] Documents submitted in a non-standard form format or which otherwise 
fail the attempt by the field extraction and OCR module (214) to extract field 

20 information can be incorporated into the system via rekeying at the manual input 
module (212). This directly produces the correct flat file format and this is then 
transferred to the duplicate check / database interface module (221) where the 
system storage is first checked to determine if the received flat file is already stored 
in the database hardware (104). 

25 [0053] The scanned forms of the documents are then are transferred to the field 
extraction and OCR module (214) where the information of each field is extracted 
and the fiat file form of the document is created. Field extraction involves the action 
of extracting information from predetermined fields and creating a "flat file" for the 
scanned document. The field extraction process requires that the documents be 

30 submitted using a predefined form (a form having defined "fields" which are located 
at predefined locations and are predetermined to hold specified information such as 
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for a last name, first name, etc.) which is available upon request or from the Internet 
website. Use of the predefined form is encouraged for ease of data incorporation 
and to ensure that all required information is presented in sufficient detail. Once the 
flat file of a document is created by the field extraction and OCR module (214), it is 
5 transferred to the duplicate check / database interface module (221). 

[0054] If the flat file already exists as a record in the database hardware (104), 
the system does not immediately store it but determines which version (stored record 
versus received flat file) is more completely filled out and will retain that copy. This 
process is described in greater detail later herein in reference to FIG. 7. 

1 0 [0055] In one embodiment, a cutoff date is defined after which no further 

submissions are accepted. In the case of mailed submissions, the postmark date is 
the date used to determine timeliness of submission. Thus, further to this 
embodiment, an extension time period (such as 10 days) is employed to allow timely 
mailed hardcopies and other mailed media to arrive at the competition system. Once 

15 it is determined that all documents relating to candidate applications have been 
incorporated into the system, the qualification and eligibility subsystem (230) 
implements basic qualification and eligibility processes. 

[0056] The qualification and eligibility subsystem (230) is shown comprising a 
completeness filter module (232), a qualification filter module (233), a partner filter 

20 module (234), and an eligibility filter module (235). 

[0057] The completeness filter module (232) is coupled to the qualification filter 
module (233). The qualification filter module (233) is coupled to the partner group 
filter module (234). The partner group filter module (234) is coupled to the 
eligibility filter module (235). The completeness filter module (232), the 

25 qualification filter module (233), the partner filter module (234), and the eligibility 
filter module (235) are all coupled to both the non-select segment interface module 
(222) and the candidate segment interface module (223). 
[0058] The completeness filter module (232) (also called the level 1 filter 
module) analyzes applicant packages for completeness. In one embodiment, the 

30 completeness filter module (232) checks applicant packages for both document 
completeness (i.e., that all required applicant documents are present) and 
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information completeness (i.e., that all required data or information is present). Any 
applicant package which fails either document completeness or information 
completeness is transferred to the non-select database and the applicant package is 
deleted from the applicant list and entered in the non-select list. The application of 
5 level 1 filters is discussed in greater detail later herein in reference to FIG. 8. 

[0059] The qualification filter module (233) (or level 2 filter module) analyzes 
candidate packages to determine if the candidates meet various minimum 
qualification standards. By way of example, these minimum standards can include a 
minimum GPA standard. 

1 0 [0060] The partner group filter module (234) (also called the level 3 filter 

module) analyzes the candidate packages to determine if the candidates meet various 
partner group- specific minimum standards. Partner groups are subpools of 
candidates (i.e., less than the full pool of candidates in the competition system) 
based on any one or more candidate-specific criteria. Partner groups are created so 

15 as to provide the ability to evaluate, rank, and select winners in ways sensitive to the 
specific partner group. By way of example, partner groups might be set up in an 
academic scholarship setting by looking to such variables as the candidates' 
geographical location, ethnic culture, poverty level, mother tongue, national origin, 
age, and so on. Thus, for a partner group for candidates having a non-English 

20 mother tongue, abilities in English might not be used during filtering as having little 
or no correlation to the academic abilities of candidates in that partner group. 
Conversely, English abilities might be stressed in the filtering of partner groups for 
which English as a mother tongue was a criteria. 

[0061] The eligibility filter module (235) (or level 4 filter module) imposes one 
25 or more heightened minimum response standards. In one embodiment for a 
competition system having multiple optional essay questions, for example, a 
minimum number of essay responses are required by the eligibility filter. 
[0062] It is noted that the use of more than one filter level or the use of partner 
groups is not necessary and having all filtering done by one software module is 
30 possible. The separation of filters into levels allows for easier use of criteria having 
different purposes. Separation of filters also allows for better separation of criteria 
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which is useful when different individuals or committees are responsible for 
different filter criteria. 

[0063] The operation of the level 2-4 filters is discussed in greater detail later 
herein in reference to FIG. 9. 
5 [0064] The evaluation and selection subsystem (240) implements evaluation of 
applicant packages and selection of competition winners. 

[0065] The evaluation and selection subsystem (240) is shown having a queue 
with reader module (241), a reader eligibility module (242), a candidate eligibility 
module (243), a scoring module (244), a read verification module (245), a phase 1 
10 selection module (246), a phase 2 selection or tie-breaking module (247), and a 
reader monitoring module (248). 

[0066] The queue with reader module (241) is coupled with the reader 
eligibility module (242). The reader eligibility module (242) is coupled to the 
candidate eligibility module (243) and the queue with reader module (241). The 

1 5 candidate eligibility module (243) is coupled to the scoring module (244) and the 

non-select segment interface module (222). The scoring module (244) is coupled to 
the read verification module (245) and the reader monitoring module (248). The 
read verification module (245) is coupled to the phase 1 selection module (246) and 
the queue with reader module (241). The phase 1 selection module (246) is coupled 

20 to the phase 2 selection or tie-breaking module (247). The reader eligibility module 
(242), the candidate eligibility module (243), the scoring module (244), the phase 1 
selection module (246), and the phase 2 selection or tie-breaking module (247) are 
all coupled to the candidate segment interface module (223). 
[0067] The queue with reader module (24 1 ) queues candidate packages with 

25 evaluators or readers for later evaluation (also called reading). In one embodiment, 
each applicant package is matched and queued with one evaluator. Matching is 
performed by comparing information from candidate packages with potential 
evaluators. In one embodiment, candidate packages are matched to evaluators by 
one or more of the following information: cultural background, ethnic background, 

30 age, or geographic residence. Each evaluator thereafter should have one or more 
candidate packages listed in the queue for that evaluator. 
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[0068] The reader eligibility module (242) requires the evaluator to pass an 
eligibility procedure for each candidate package prior to evaluation. This process 
requires the evaluator to answer one or more evaluator/reader eligibility questions 
with respect to a particular candidate package. The reader eligibility module (242) 
5 determines from the evaluator' s responses whether the evaluator is able to evaluate 
the candidate package or not. In one embodiment, for example, evaluator questions 
can be concerned with whether the evaluator attended the same school as the 
candidate or has other connections with the candidate which could raise questions of 
bias. If the evaluator is unable to evaluate the candidate, the evaluator is blocked 
1 0 from evaluating the candidate and the candidate package is requeued with another 
evaluator at the queue with reader module (241). 

[0069] The candidate eligibility module (243) requires the evaluator to 
determine whether the candidate is eligible. This determination can be simply a 
verification check of the same requirements enforced by the filters in the 
1 5 qualification and eligibility subsystem (230) but additional or alternative 
requirements can be applied by this module. 

[0070] The scoring module (244) implements the actual evaluation of applicant 
packages by evaluators. An embodiment for an algorithm implementing evaluation 
or reading of candidates is discussed in detail later herein in reference to FIG. 1 1 . 

20 The read verification module (245) verifies that the candidate package was evaluated 
by the required number of evaluators. For example, in one embodiment, each 
candidate package must be evaluated by two evaluators, thus candidate packages 
having fewer than two evaluations are requeued with another evaluator at the queue 
with reader module (241). 

25 [0071] The phase 1 selection module (246) handles initial selection of candidate 
finalists who, if confirmed at the confirmation subsystem (250) described 
hereinafter, become competition winners. The phase 2 selection or tie-breaking 
module (247) handles the second phase or stage of candidate finalist selection. The 
phase 2 selection module (247) employs a finer selection paradigm than the phase 1 

30 selection module (246) and is used to select between candidates who are very similar 
in overall evaluation scores. An algorithm for the phase 1 selection module (246) is 
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discussed in detail later herein in reference to FIG. 12a and an algorithm for the 
phase 2 selection module (247) is discussed in detail later herein in reference to FIG. 
12b. 

[0072] The reader monitoring module (248) implements monitoring of the 
5 evaluators during evaluations and can terminate an in-progress evaluation or nullify 
a completed evaluation if a problem is detected. Monitoring can take several forms. 
In one embodiment, evaluator responses are compared against corresponding 
reference standards. For example, if a competition involves evaluating candidates 
over several criteria such as writing skill and other criteria, monitoring will involve 

1 0 comparing the evaluator's responses for these criteria to standards. The evaluator's 
response to writing skill, for example, can be compared to the average score the 
evaluator has given to writing skill in all previous evaluations completed by that 
evaluator. Another method is to compare the evaluator's responses to a 
predetermined standard or to the average across all evaluators. If an evaluator is 

1 5 found to be erratic, corrective measures such as imposing a break, counseling, or so 
forth can be mandated. Additionally, completed evaluations found to be erratic can 
be nullified and the candidate package requeued with another evaluator. 
[0073] The database interfacing subsystem (220) provides the necessary 
interfacing between the database hardware (104) and the rest of the system. The 

20 database interfacing subsystem (220) comprises a duplicate check / database 

interface module (221), a non-select segment interface module (222), a candidate 
segment interface module (223), and a competition-winner segment interface 
module (224). 

[0074] The duplicate check / database interface module (221) is coupled to the 
25 database hardware (104), the network interface module (21 1), the manual input 
module (212), the field extraction and OCR module (214), and the completeness 
filtering module (232). The duplicate check / database interface module (221) 
supplies and retrieves flat files from the database hardware (104). Additionally, the 
duplicate check portion of the duplicate check / database interface module (221) 
30 determines whether a flat file to be stored already exists in the database hardware 

(104) as a record, and if so, determines which of the two are to be stored as the most 
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complete version and which is to be "set aside" (i.e. stored but indicated as a 
rejected duplicate). The duplicate check / database interface module (221) also, 
during the storing of flat files, determines if any other records in the database (104) 
relate to the same applicant as the file to be stored, and if so, they are associated 
5 together. For example, in systems requiring an applicant package to consist of an 
applicant form, a nominator form, and a recommender form, the duplicate check / 
database interface module (221), when storing a document, will check if the other 
two forms are present for the form being stored, and if one or both is, the forms will 
be associated together as an applicant package. An algorithm for implementing the 
1 0 duplicate check / database interface module (221) is discussed in greater detail in 
reference to FIG. 7. 

[0075] The non-select segment interface module (222) is coupled to the 
completeness filter module (232), the qualification filter module (233), the partner 
group filter module (234), the eligibility filter module (235), and the candidate 

1 5 eligibility module (243). The non-select segment interface module (222) interfaces 
other software modules with the non-select segment (112) which holds candidate 
packages of candidates who ultimately do not become competition winners. 
[0076] The candidate segment interface module (223) is coupled to the 
completeness filter module (232), the qualification filter module (233), the partner 

20 filter module (234), the eligibility filter module (235), the reader eligibility module 
(242), the nominee eligibility module (243), the scoring module (244), the read 
verification module (245), the phase 1 selection module (246), and the phase 2 
selection or tie-breaking module (247). The candidate segment interface module 
(223) interfaces other software modules to the candidate segment (114) for access to 

25 candidate packages which have not been marked as non-selected. 

[0077] The competition- winner segment interface module (224) is coupled to the 
phase 1 selection module (246), the phase 2 selection module (26), the confirmation 
module (252) (via the connector "C"), and the re-qualification module (263) (via the 
connector "D"). The competition- winner segment interface module (224) interfaces 

30 other software modules to the competition- winner segment (116) for storage of and 
access to candidate packages marked as competition winners. 
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[0078] Referring to FIG. 3b, shown is part 2/2 of a block diagram in greater 
detail of the software architecture of FIG. 2. 

[0079] Shown are a confirmation subsystem (250), a tracking and re- 
qualification subsystem and a payment and correspondence generation subsystem 
5 (270). Each of these subsystems has various modules that are discussed in greater 
detail as follows. 

[0080] The confirmation subsystem (250) is shown having a confirmation 
document request module (251) and a confirmation module (252). The confirmation 
document request module (251) is coupled to the confirmation module (252) and the 

1 0 correspondence printing interface module (271). The confirmation module (252) is 
coupled to the correspondence printing interface module (271), the check printing 
interface module (272) and the competition winner tracking module (261). The 
confirmation subsystem (250) implements confirmation of candidate finalists. 
[0081] The confirmation document request module (251) determines what 

1 5 documents are needed to confirm a candidate finalist and handles the generation of 
correspondence to request these documents. The confirmation module (252) 
implements the actual confirmation of candidate finalists and handles the generation 
of award or rejection letters and, in the case of winners, initial check issuance if 
appropriate. 

20 [0082] An exemplary algorithm for the operation of the confirmation subsystem 
(250) is discussed in greater detail later herein with respect to FIG. 13. 
[0083] The tracking and re-qualification module (260) is shown having a 
competition winner tracking module (261), a re-qualification document request 
module (262), and a re-qualification module (263). 

25 [0084] The tracking and re-qualification module (260) implements progress 
tracking of competition winners and submits the competition winners to re- 
qualification to ensure that they continually meet any requirements of the 
competition system (100). For example, in one embodiment for providing academic 
scholarships, competition winners (scholars) have defined academic requirements in 

30 order to continue receiving scholarship award payments. The competition winner 
tracking module (261) is coupled to the re-qualification document request module 
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(262). The re-qualification document request module (262) is coupled to the re- 
qualification module (263) and the correspondence printing interface module (271). 
The re-qualification module (263) is coupled to the correspondence printing 
interface module (271), the check printing interface module (272), the competition- 
5 winner segment interface module (224), and the document receiving module (215). 
[0085] The competition winner tracking module (261) determines when re- 
qualification or progress documents or information is needed. The re-qualification 
document request module (262) handles the scheduling and printing of 
correspondence to the necessary recipients which request the submission of required 

1 0 documents or other information so that competition winner tracking and re- 
qualification can be carried out. The re-qualification module (263) implements the 
actual re-qualification evaluation of competition winners. 
[0086] An exemplary algorithm for the tracking of competition winners is 
discussed in greater detail later herein in reference to FIG. 14. 

1 5 [0087] The payment and correspondence generation subsystem (270) is shown 
having a correspondence printing interface (271) and a check printing interface 
module (272). 

[0088] The payment and correspondence generation subsystem (270) 
implements generation of checks and correspondence. The correspondence printing 

20 interface (271) is coupled to the confirmation document request module (250), the 

confirmation module (252), the re-qualification document request module (262), and 
the re-qualification module (263). The check printing interface module (272) is 
coupled to the confirmation module (252) and the re-qualification module (263). 
[0089] The correspondence printing interface (271) interfaces with 

25 wordprocessors or other end devices such as the printer (1 50) to print 

correspondence necessary for implementation of any other processes in the system. 
The check printing interface module (272) enables the printing of competition award 
checks such as on the printer (152). 

[0090] The reporting module (280) is shown with multiple interconnections 
30 radiating outward indicating that it is coupled with substantially every other module 
shown in FIGS. 3a and 3b. The reporting module, as previously discussed, collects 
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information and generates reports. As an example of the detail which the reporting 
module (280) collects, the reporting module (280) is coupled to the queue with 
reader module (241) from which it determines each evaluator that an applicant 
package is queued to. From the reader eligibility module (242), the reporting 
5 module (280) documents the results of the reader evaluation, and so on. From these 
data, the reporting module (280) can generate reports documenting exactly what 
occurred with a particular applicant package as well as the performance of a 
particular reader, for example. 

[0091] The external systems interface module (290) is likewise coupled to all 
1 0 modules necessary in FIGS. 3a and 3b. As discussed previously herein, the external 
systems interface module (290) interfaces the modules and subsystems shown in 
FIGS. 3a and 3b with external systems such as finance systems, human resources 
systems, and so forth. 

[0092] Referring to FIG. 3c, shown is the block diagram of FIG. 3a and the 
1 5 reporting module (280) showing interconnections of the reporting module (280). 
Similarly, referring to FIG. 3d, shown is the block diagram of FIG. 3b showing 
interconnections of the reporting module (280). 

[0093] Shown in FIG. 3c are all of the modules shown in FIG. 3a and the 
reporting module (280). Shown in FIG. 3d are all of the modules of FIG. 3b. The 

20 reporting module (280) is shown in both figures connecting with all the shown 
modules. In operation, the reporting module (280) records the events of each 
module for documentation purposes. The reporting module (280) is able to 
synthesize a great variety of reports from the recorded information including, but not 
limited to document submission error reports, scanning error reports, possible 

25 duplicate reports, lists of all applicants for a specified competition, lists of all floater 
documents, status reports on a specified applicant package or a specified set of 
applicant packages (such as by an entered list or a specified partner group or a group 
defined by a specified piece or range of information), evaluation summary reports by 
applicant, partner group, evaluator, scholar financial reports, scholar progress 

30 reports, data analysis reports, trend analysis reports, and so forth. The reporting 
module also stores and can reference in reports the documenting of decisions by 
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competition system personnel. Such documentation provides proof, if later required, 
of the decisions made by system administrators and personnel. Decisions which 
may be documented in this way include decisions regarding duplicate document and 
fraud situations, the order of evaluation variable application for tie-breaking, what 
5 constitutes a complete document, and so forth. It is noted that one of ordinary skill 
in the art, in implementing a competition system ( 1 00) for a defined purpose and a 
defined reporting capability, would know what modules to couple to the reporting 
module and what data the reporting module would accumulate and store. 
Additionally, one of ordinary skill in the art, given a defined purpose and a defined 
1 0 reporting capability, would know which modules the external systems interface 

module (295) would need to be coupled to and what transfer protocols would need to 
be implemented. 

[0094] Referring to FIG. 4, shown is a general flowchart of the steps traversed in 
operation of the systems of FIGS. 1 and 2. 

1 5 [0095] The competition system (100) receives (402) application documents 
which can originate from applicants themselves and/or other individuals. In one 
embodiment, as discussed previously, are applications and documents in support 
thereof from nominators or recommenders. These submissions can be via electronic 
means such as, but not limited to, electronic submissions via the network (122) such 

20 as via the Internet, or via hardcopy submissions such as paper copies submitted by 
postal mail (130). In the case of hardcopy submissions using a preformatted form 
designed for the system of the present invention, the hardcopy is scanned using a 
scanning system (132). During the scanning process, form fields are identified and 
the information contained therein is extracted and read. In one embodiment, the 

25 scanning is done by optical means using a commercially available scanner although 
any other method, such as by mechanical means, can be implemented. 
[0096] In one embodiment, regardless of method of submission, forms and 
documents must be converted to a flat file format discussed previously in reference 
to FIG. 2. In submissions via the network (122), the flat file is created directly from 

30 the inputted information. In submissions incorporated via the scanning system 

(132), the scanning system produces the flat file from the data retrieved during field 



26 



Attorney Docket 72156 



extraction and optical character recognition (OCR). Similarly, flat files are 
generated from electronic submissions on computer-readable media or manual entry 
via a workstation such as workstation (140). 

[0097] Once the flat file form of received documents are present, the system 
5 stores (404) the flat files in the database hardware (104) as records and they are 
marked as being in the staging segment (110). At the time of storing, the system 
groups (404) or associates the flat file with any other records from the same 
applicant. It is at this stage that the various records relating to the same applicant are 
associated to form applicant packages. While it is possible that duplicates of one or 
1 0 more of these forms can be submitted to the system, only one of each form type is 
included in an applicant package. Any duplicate forms are retained but identified as 
duplicates (the process of identifying the duplicate forms and excluding them from 
the candidate packages is termed "setting aside"). 

[0098] In one embodiment, the association of records is done by applicant social 

1 5 security number. In this embodiment, the applicant record (the application 

submitted by the applicant his- or herself) is the key file (or key record). Once the 
applicant form is stored as a record, the applicant identified in the record is added to 
an applicant list. If any other required forms identifying that applicant are later 
submitted, such as a nominator form or a recommender form, they are associated 

20 with the corresponding applicant record or key file. Required forms such as 
nominator forms or recommender forms, which are submitted without a 
corresponding key form (the applicant form) being present are termed "floaters" 
indicating they do not have a key file to be associated to. 
[0099] After the cutoff date (406), further electronic submissions via the 

25 network interface module (211) are prevented. The cutoff date is chosen so as to 
provide enough time to carry out the rest of the competition processing prior to any 
necessary deadlines. In one embodiment, if hardcopy forms submitted by mail or 
other means are allowed in a competition, the postmark dates of mailed submissions 
are used to determine whether a submission was timely. When using postmarked 

30 dates to determine cutoff timeliness, extra time is allowed beyond the cutoff date 

(such as 10 days) to allow for the physical delivery of the mailed submissions to the 
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competition processing site and the necessary subsequent incorporation of the 
hardcopy data into the competition system. All applicants identified in any form 
which was received by the cutoff date and the postal mail extension time period 
allowed for are identified and recognized within the system and will therefore be 
5 considered during the competition. 

[0100] At this point, the competition system checks (408) the completeness of 
the applicant packages. This determination is also referred to as level 1 filtering. In 
one embodiment, the filtering is done in two or more stages or levels which are 
described in greater detail in reference to FIG. 8 and FIG. 9. In one embodiment, the 

10 completeness of applicant packages is checked in two ways. One determination is 
whether an applicant package has document completeness. As discussed previously 
in reference to one embodiment, a complete applicant package is comprised of an 
applicant or nominee form, a nominator form, and a recommender form. Any 
package determined not to contain all required initial forms is determined to be 

15 incomplete. A second determination is whether each document in a candidate 

package contains all required data or information. As each document required in a 
complete candidate package is different from all others, each document generally has 
different required data or information standards. Applicant packages which are 
lacking one or more required documents or which contain documents missing one or 

20 more required datum or pieces of information are determined to be incomplete. 

These applicant packages are redesignated as non-selects and are removed from the 
applicant list, added to the non-select list, and, along with any duplicate forms, are 
marked to be in the non-select segment (112). All complete applicant packages 
which meet the basic eligibility requirements are marked to be in the candidate 

25 database (114). Thereafter, these applicant packages are referred to as candidate 
packages. Candidate package are then filtered (412) with one or more higher level 
filters. 

[0101] After candidate packages pass the filtering process, they are read (414). 
Reading involves evaluation of the candidates with respect to the rubric of the 
30 competition system. The term rubric refers to the variables and the scoring process 
which are used to score or select the winning candidate. The variables can be 
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objective or quantitative (such as grade point average or GPA) or subjective or 
qualitative (such as the results of essay evaluation). The variables can also include 
both cognitive variables (GPA, essay writing, etc.) or non-cognitive variables 
(leadership experience, presence of a support person, etc.). In one embodiment, 
5 evaluation or reading is done by humans who are trained to evaluate the candidate 
packages, although automated reading implemented by software means or a 
combination of automated processes and human evaluation can be implemented. In 
one embodiment, human readers evaluate the candidate packages against a rubric 
composed in the greater part by non-cognitive variables such as, but not limited to: 

1 0 positive self-concept, realistic self-appraisal, understanding/navigation of social 
systems, preference for long-term goals over short-term needs, availability of a 
support personage, leadership experience, community service, and non-scholastic 
knowledge concentration (i.e. appreciable knowledge acquisition about a field not 
covered in past academic environments). 

1 5 [0102] Evaluation or reading is generally done multiple times for each candidate 
package for reasons of reducing the effects of spurious read deviations. In one 
embodiment, the reading process is completed two independent times for each 
candidate package (i.e., the candidate packages are each evaluated by two different 
readers). In other embodiments, however, multiple partial reads are used which 

20 allows expert readers to specialize on a subset of evaluation variables. In still other 
embodiments, only one read or more than two reads can be used per candidate 
depending on the repeatable accuracy of readers, the limitations of the competition 
finances, or the fairness safety factor desired. The process of evaluation or reading 
is described in greater detail later herein in reference to FIG. 10. 

25 [0103] In order to ensure the accuracy and repeatability of human (and 
automated) readers, reader normalization (or reader norming for short) is 
implemented in one embodiment. Reader normalization refers to those processes 
implemented to ensure readers operate within a predetermined variance standard 
during their evaluation. Reader normalization is used during training and continued 

30 during the actual reading of candidates. As a component of reader norming during 
actual evaluations, reader monitoring is implemented which allows for detection of 
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errant evaluations or biases. Reader norming is discussed in greater detail later 
herein in reference to FIG. 1 1 . 

[0104] Once the candidate packages have been read, the candidate finalists are 
selected (416). Selection generally is implemented by selecting the candidates with 
5 the highest read scores. The selection process can be complicated and involved 
when the number of competition winners is predetermined and the number of read 
variables is relatively low because generally there will be a read score for which 
some candidates will be selected, but due to the limited number of total competition 
winners, others at the same score will not be able to be selected. The selection 

1 0 process is described in greater detail later herein in reference to FIGS. 12a and 12b. 
[0105] Once the candidate finalists are determined (416), the candidate finalists 
are put through a confirmation process (4 1 8) which involves verifying statements 
made in the candidate's application (the nominee form in an embodiment) and any 
other required documentation (such as the nominator and recommender forms, in an 

1 5 embodiment) and also verifying that the candidate meets any eligibility requirements 
the data for which were not known at the time of application. The process of 
confirmation is aided by the reception (420) of required confirmation documents. 
By way of example, in an academic scholarship competition it may be that the 
candidate does not know which college or university he or she will attend at the time 

20 of application, but this information must be known and supplied prior to the 

academic scholarship grant. The process of confirmation is discussed in greater 
detail later herein with respect to FIG. 1 3 . 

[0106] After confirmation, competition winners are tracked (422) to monitor 
their progress and, optionally, to determine whether or not the competition winner is 
25 maintaining any post-competition requirements. In an academic scholarship system, 
for example, post-competition requirements may relate to academic performance, 
and if met, the system responds by continuing payment of the scholarship award. 
The post-competition tracking system is discussed in greater detail later herein in 
reference with FIG. 14. 
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[0107] Referring to FIG. 5, shown is a flowchart of an exemplary algorithm 
(500) for the steps traversed in the operation of electronic document submission of 
FIG. 4. 

[0108] As one method of receiving information from potential candidates, a 
5 website is maintained which allows access and online information submission by 
individuals. When an individual accesses (502) the website, for security reasons, the 
website requires (504) identification of the accessing individual by a logon protocol. 
Additionally, in one embodiment, security is implemented during the submission 
process. By way of example, digital signatures are used and retained by the system 
1 0 for verification. Alternatively, the system can require or allow the use of personal 
identification smart cards to verify identity. 

[0109] Once the individual is successfully logged on, the system will provide the 
individual with options which include online form completion and submission. Fhe 
individual then indicates his or her role (506) (e.g., in one embodiment as applicant, 

1 5 nominator, or recommender) and, in the case of individuals providing support 

documents, the identity of the applicant (504) for which they wish to submit a form. 
After selecting online form creation, the system which first determines (508) 
whether that form was already submitted (i.e. whether that individual already 
submitted a form for the candidate identified). Duplicate submissions are sometimes 

20 requested for various reasons such as the individual wishes to change some 

information, the individual forgets the prior submission, and so on. In order to 
prevent creation of duplicate documents, the system prevents (5 1 0) the individual 
from continuing if the form requested has already been submitted to the system. 
[0110] In one embodiment, duplicate forms are detected using the candidate's 

25 social security number only. In another embodiment, potential duplicate 

applications are detected by the use of the last name, the first letter of the first name, 
the date of birth, and the state of birth and/or residence. In this embodiment, 
potential duplicates identified by these criteria are included in a duplicate report and 
system administrators will be able to access a special interface set up to allow 

30 simultaneous display of data from two applications so that the two potentially 

duplicated forms can be viewed together. Duplicates can arise which have alternate 
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candidate social security numbers. This can occur in the case of a duplicate form 
being input with an erroneous candidate social security numbers or intentionally 
changed social security number (i.e., fraud). 

[0111] Detection of both innocent and intentional duplication of forms is 
5 necessary for proper competition administration. Cases of apparent duplicates with 
different social security numbers do often occur and a common situation is the case 
of twins. By way of example, it is noted that in some Asian cultures, twins may 
receive identical or nearly-identical names and thus may appear to be a highly likely 
case of duplication when in actuality it is not. In one embodiment, the system 

10 administrator reviews all possible duplicate situations prior to the involved 

documents continuing in the competition process. To facilitate this, the system 
generates a possible duplicate report such as shown in FIGS. 22-23. 
[0112] If the form requested was not previously completed, the individual is 
presented (530) with the appropriate form for online completion. In the embodiment 

] 5 shown, the individual is interacting directly with the system, so the system must wait 
(530) for each field submission (usually by pressing of the enter key or the tab key). 
In another embodiment, forms can be completed by use of software such as a 
downloaded program or browser plugin and the results submitted at completion. 
[0113] Next, the system determines (512) whether the submit button (indicating 

20 the individual has completed the form and wishes to submit the form for storage). If 
the individual's action was not submission but information entry in a field, the 
system determines (5 1 4) whether the entered data meets basic validity criteria for 
that field and presents (5 1 6) any error message to the individual. By way of 
example, if the field requires the birthdate of the candidate, the system may impose a 

25 general form requirement that the date be entered in the common mm/dd/yy form 
(i.e. two digit month numeral followed by two digit day numeral followed by a two 
digit year numeral). The system could also provide a presumed validity requirement 
by presuming invalid any birthday which is in the future, was in the past 10 years, or 
was prior to 1 00 years previously (under the hypothesis that no candidate would be 

30 younger than 10 years old or greater than 100 years old). If an error message is 

presented, the cursor resets (5 1 8) in the field where the invalid data was entered and 
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the system again waits (532) for information entry and thereafter continues as 
discussed previously. 

[0114] If a field submission is determined (514) valid, the system automatically 
positions (520) the cursor in the next field and thereafter waits (532) for further 
5 information entry and thereafter continues as described previously. 

[0115] When the individual has completed the form and selects the submit 
button, the system determines (522) whether the form is completely filled out. If 
not, the system allows (524) the saving of incomplete forms for later continuance 
after which the online form entry session is finished for now as shown by done 
1 0 indicator (526). If the form is determined (522) to be complete, the form is stored in 
the database hardware (104) as a record and marked (528) to be in the staging 
segment (110) and the form entry session is finished as shown by done indicator 
(526). 

[0116] Referring to FIG. 6 shown is a flowchart of an exemplary algorithm (600) 
1 5 of the steps traversed in the operation of hardcopy submissions in the systems of 
FIGS. 1-2. 

[0117] Another way besides electronic submission for forms to be submitted by 
individuals is by paper or hardcopy submission such as by postal mailO. Other forms 
of submission could also be used which include, but are not limited to, facsimile 
20 submission, compact disc or other optical recording device submission, electronic 
file submission other than online form completion such as by a portable document 
format (PDF) file, submission on magnetic media such as tape or disc (e.g., floppy 
disc or harddisk), and mechanical submission such as by punched card, sheet, or 
tape. 

25 [0118] As the competition system (100) receives (602) submissions, they are 

collated (604) by form type (e.g. by applicant, nominator, and recommender forms as 
discussed previously for one embodiment). At periodic intervals, or as convenient, 
the received forms are scanned (606) in batches. After scanning, the resultant 
images are submitted (6 1 0) to a field extraction and optical character recognition 

30 (OCR) process to extract the data in the images. The system then determines (612) 
whether the field extraction and OCR process was successful, and if it was, the 
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algorithm (600) then creates (614) an export file and then is done for the data 
extraction of that image as indicated by the done indicator (616). After this, the 
algorithm (600) generally selects (602) another form for processing and continues as 
before until all scanned documents are likewise submitted to the field extraction and 
5 OCR process. If the field extraction and OCR process is determined (612) to have 
not been successful, forms can be manually entered (618). The algorithm (600) then 
creates (614) an export file and then is done for the data extraction of that image as 
indicated by the done indicator (616). 

[0119] Generally, errors sometimes result in interpretation of field information. 

10 By way of example, an error could be as simple an occurrence as a comma having 
been entered when a "/' was required (such as in a date field). A scan report is 
generated which summarizes the results of the scan process of each batch and if 
errors resulted, these are indicated by document in the scan report. 
[0120] Referring to FIG. 7, shown is a flowchart of an exemplary algorithm 

1 5 (700) of the steps traversed in the storing and grouping of records shown in block 
(404) of FIG. 4. 

[0121] The algorithm (700) stores flat files into the database hardware (104) as 
records. These flat files are received through any of the methods discussed 
previously in reference to FIGS. 5-6 (such as electronically and via postal mail with 

20 subsequent scanning or manual entry). The algorithm (700) begins by determining 
(702) the form type (i.e. in one embodiment, whether the flat file represents a 
applicant form, nominator form, or recommender form) and applicant identity from 
the flat file. In one embodiment this is easily done as forms must be submitted using 
predefined forms having specified fields or identifying information or markings 

25 which identify the nature of the form. Next, the algorithm (700) determines (703) 
whether any records in the database (104) for applicants with a different social 
security number have the same form type and contain data which substantially 
matches the flat file to be stored. If any records are found substantially matching the 
file to be stored, this is a potential fraud situation and an entry is made (704) in the 

30 duplicate list. 
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[0122] Next, the algorithm (700) determines (705) if a record exists in the 
database (104) which is a match for the form type and the same applicant. If not, the 
flat file is stored (706) in the database hardware (104) as a record. If the form is an 
applicant form (708), it is indexed (7 1 0) in the applicant list or key list. The 
5 applicant form or key list is The algorithm (700) then checks a floater list for any 
floaters corresponding to the stored applicant record and if there are any, they are 
associated (712) with the just-stored applicant record. The floater list is a list of all 
forms (such as the nominator and recommender forms in the prior embodiment) for 
which a corresponding applicant record has not been stored in the database hardware 
10 (104). The algorithm (700) is then done for this flat file as indicated by the done 
indicator (714). 

[0123] If the flat file is not an applicant form (708), a check (716) is made if the 
corresponding applicant (or key) form has already been indexed. This is done in one 
embodiment by use of the candidate's social security number (SSN), which is also 

1 5 required on the nominator and recommender forms in identifying the applicant for 
which the form is submitted. If a corresponding applicant form has not been 
received by the competition system (100), the file is indexed (720) and flagged as a 
floater in the floater list. After indexing of the file in the floater list, the algorithm 
(700) is done with this form as shown by the done indicator (714). 

20 [0124] If a corresponding applicant form is found (7 1 6) to have been indexed by 
the system, the stored record is associated (718) with the corresponding applicant or 
key form. The algorithm (700) is then done with this form as shown by the done 
indicator (714). 

[0125] If the flat file is determined (705) to already be in the database hardware 
25 (104), this means that the flat file is considered a possible duplicate and is indexed in 
a duplicate list. The duplicate list is a report which keeps track of all duplicates. 
The algorithm (700) next compares the stored record with the received flat file to 
determine (722) whether the received flat file is more complete than the record 
already stored in the database hardware (104). If it is, the record already stored in 
30 the database hardware (104) is flagged as "set aside" and the received file is stored 
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and is the record which is used by the competition system (100) during further 
considerations. 

[0126] It is noted that as discussed previously, by consideration of the duplicate 
list, system administrators or other officials can verify whether the correct action 
5 was taken. 

[0127] If the received file is determined (722) to be less complete than the record 
stored already, the received form is stored (724) in the database hardware (104) and 
flagged as "set aside", but is still associated (726) with the other documents of the 
same applicant. The algorithm (700) is then done with this form as shown by the 
1 0 done indicator (714). 

[0128] Referring to FIG. 7a, shown is a flowchart of an exemplary algorithm for 
duplicate detection review in one embodiment. 

[0129] In one embodiment, system administrators or other authorized personnel, 
must review all possible duplicate detections prior to the qualification and eligibility 

1 5 phases. A reviewer accesses the possible duplicate list to determine the possible 

duplicate situations to review. After selection (752) of a possible duplicate situation 
to review, the reviewer notes (754) the nature of the situation. If the situation is one 
of two or more documents identifying the same applicant social security number 
(SSN), the reviewer reviews the involved documents to determine which document 

20 is more completely filled out. In one embodiment, the duplicate detection / database 
interface module (291) makes an initial decision as to which document is more 
complete. In this embodiment, if the reviewer determines (756) that the duplicate 
detection / database interface module (291) made the right decision, documents 
(758) that decision and is done for that situation. Otherwise, if the reviewer 

25 determines (756) that the decision of duplicate detection / database interface module 
(291) was in error, the reviewer corrects the decision and documents (758) the 
decision and action and is done for that situation. 

[0130] If the situation is determined (754) to be a situation which does not 
involve identical social security numbers, the reviewer notes (762) whether it is a 
30 situation of possible fraud. Possible fraud situations result from two or more 

documents bearing demographic and other information which appears to refer to an 
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identical individual but which identifies different social security numbers. If the 
situation is one of possible fraud, the reviewer reviews (764) the involved 
documents and makes a determination. Possible determinations include (1) that the 
involved documents relate to different individuals (i.e. no fraud), (2) that the 
5 documents refer to the same individual (i.e. actual fraud), (3) that the documents 
refer to the same individual but that one or more social security numbers were 
improperly entered, and so forth. After the determination and resultant action, the 
reviewer documents (766) the determination and action and is thereafter done (760) 
for the situation. 

10 [0131] If the reviewer determines (762) that the situation is something other than 
simple duplicate detection or possible fraud, the reviewer reviews (768) the 
situation, makes a determination, implements the determination, and documents 
(720) the determination and implementation. Thereafter, the reviewer is done (760) 
with the situation. 

15 [0132] Referring to FIG. 8, shown is a flowchart of an exemplary algorithm 

(800) of the steps traversed in the determination of candidate package completeness 
in FIG. 4. 

[0133] Once the cutoff date (406) and any extended time for postal delivery has 
been reached, applicant packages are analyzed for completeness. In one 

20 embodiment, applicant packages are analyzed for both document completeness and 
each document is analyzed for data completeness. The algorithm (800) begins by 
selecting (802) an applicant package for analysis. The algorithm (800) determines 
(804) first whether the application package has data completeness, and if so, the 
algorithm (800) selects a form from the package. Applicant packages, to have 

25 document completeness, must be made up of a minimum required set of document 

records. For example, in one embodiment, a candidate package requires an applicant 
form, a nominator form, and a recommender form to be complete. The algorithm 
(800) then determines (808) whether the form has data completeness. For an 
applicant package to have data completeness, each document or record in the 

30 applicant package must meet a minimum data completeness standard (such as all 
required fields having non-null entries or all required data present). Note that 
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because each document in an applicant package is different, each may have different 
minimum data completeness requirements. If the record selected does have data 
completeness, the algorithm (800) determines (810) whether any more records in the 
applicant package need to be checked for data completeness, and if so, the algorithm 
5 (800) selects (806) another record and continues as described previously. If there are 
no further records to be analyzed (810), the algorithm (800) is done for this package 
as indicated by the done indicator (812). 

[0134] If a record is determined (808) not to have data completeness, it is 
flagged (814) as incomplete and the applicant package is marked (816) to be in the 
1 0 non-select segment (110). Thereafter, the algorithm (800) is done for this package 
as indicated by the done indicator (812). 

[0135] If the applicant package is determined (804) to be lacking in one or more 
records, the applicant package is flagged (818) as incomplete and designated (816) 
as a non-select by being marked as in the non-select segment (110) and being listed 
1 5 in the non-select list. Thereafter, the algorithm (800) is done for this package as 
indicated by the done indicator (812). 

[0136] Referring to FIG. 9, shown is a flowchart of an exemplary algorithm 
(900) of the steps traversed in the level 2-3 filtering of candidate packages shown in 
FIG. 4. 

20 [0137] After the candidate packages have passed the level 1 filters of data 

completeness and form completeness, they are filtered by one or more higher level 
filters. In one embodiment, a candidate package is filtered (902) by level 2 filters 
which ensure candidates meet certain qualifying characteristics. Further to this 
embodiment these qualifying characteristics include a minimum GPA score or 
25 completion of a general equivalency degree (GED) and minimum financial 

requirements (such as a maximum family income to be eligible for receiving an 
academic scholarship). The algorithm (900) then determines (904) whether the level 
2 filter qualifications are met and if they are not, the candidate package is marked 
(906) to be in the non-select segment (112) and the candidate is listed in the non- 
30 select list. 
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[0138] Otherwise, if the algorithm (900) determines (904) that the level 2 filter 
qualifications are met, further filters can be applied to the candidate package. In one 
embodiment, a candidate package is filtered (908) by level 3 filters which ensure that 
the candidate meets certain heightened response requirements. Further to this 
5 embodiment, the level 3 filters are applied specific to the candidate's subgroup or 
partner group. In this embodiment, once candidates pass the level 1 and level 2 
filters, they are grouped together into two or more subgroups or partner groups. In 
one embodiment, the grouping is done by the creation of partner lists. The grouping 
can be done based on any one or more pieces of data. By way of example, grouping 
1 0 could be done in an academic scholarship setting by looking to such variables as the 
candidates' geographical location, ethnic culture, poverty level, mother tongue, and 
so on. The level 3 filters are then be applied by subgroup/partner group and ensure 
minimum qualifications designed in light of the specific characteristics of the 
particular subgroup. 

1 5 [0139] In one embodiment, group or partner administrators are able to apply the 
level 3 filters more than one time in order to reduce the candidate pool for that 
candidate group or partner group to predetermined levels selected as optimum prior 
to going into the read or evaluation process. Criteria applied during level 3 filtering 
includes, in one embodiment, a minimum number of academic awards, a minimum 

20 number of public awards, a minimum number of honors, a minimum number of 
leadership roles, a minimum number of student excel bubbles completed, a 
minimum class rigor average, a minimum amount of community service, a 
maximum amount of personal circumstance exceptions, and a minimum number of 
paid hours of employment. 

25 [0140] In another embodiment, the level 3 filters, in order to effectively reduce 
the pool of candidates to a desirable number, are predetermined. In this 
embodiment, prior to the start of the competition process, a predetermined set of 
discretionary criteria and an ordering of their application is determined. 
Additionally, a desired pool size or a target range within which the pool must 

30 number is selected. Thereafter, in the actual filtering process, the discretionary 

criteria is applied in increments and the number of applicants remaining in the pool 
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is ascertained after each increment, with the filtering stopping when the target pool 
size is reached. By way of example, the first 1 0 discretionary criteria in one 
embodiment are, in a competition for employment at a law firm, a law degree from a 
2 nd tier school or higher, a grade point average of 3.2 or higher in a law degree, 
5 completion of one or more engineering degrees, a grade point average of 3 .2 or 

higher for one engineering degree, the attendance of a top 20 engineering school for 
one engineering degree, a law degree from a 1 st tier school, a grade point average of 
3.5 or higher in a law degree, completion of two or more engineering degrees, a 
grade point average of 3.5 or higher in an engineering degree, and the attendance of 
10 a top 10 engineering school for one engineering degree. Thus, these criteria would 
be applied in order until the pool of applicants was reduced to the desired level. 
[0141] In another embodiment, the level 3 filters, in order to effectively reduce 
the pool of candidates to a desirable number, are determined as a function of the 
number of applicants in the applicant pool. In this embodiment, the discretionary 
1 5 criteria is predetermined but instead of their application being dependent on a 

required outcome, their application is dependent on the initial size of the applicant 
pool. In this variation, the criteria itself can be dependent on the size of the pool of 
candidates. For example, again in a system for the selection of a law firm associate, 
one criteria for a pool of applicants below 50 is that the applicants must have a 3.2 
20 grade point average or higher in a law degree and for a pool of 1 00 or more 

candidates, the criteria is a minimum of a 3.5 grade point average. Alternatively, the 
criteria could be that the applicants must have a law school grade point average 
which exceeds 4 - (1 / x) where x is the number of applicants in the pool. After 
either of the block (910) or the block (906), the algorithm (900) is finished as shown 
25 by done indicator (912). 

[0142] If the candidate package passes (910) the level 3 filters, the candidate 
package has passed the filtering phase of the competition as shown by the done 
indicator (912). If the candidate package does not pass (910) the level 3 filtering, it 
is stored (906) in the non-select database (112) and the candidate is listed in the non- 
30 select list. 
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[0143] Additional or fewer levels of filtering can be applied within the scope of 
the present embodiment. In one variation, an additional level 4 set of filters is 
applied which ensure that the candidate has responded to a minimum number of 
essay questions in his or her applicant form. The system applies the level 4 filters in 
5 a manner similar to that as described for the level 2 and level 3 filters. 

[0144] Referring to FIG. 10, shown is a flowchart of an exemplary algorithm 
(1000) of the steps traversed in operation of the reading process in FIG. 4. 
[0145] The system begins by queuing (1002) a applicant package with an 
evaluator (also referred to as a reader) who has not previously evaluated that 

1 0 applicant package or been disqualified from evaluating or reading that applicant 
package. The evaluator then begins by responding (1004) to a set of reader 
qualification questions which document whether the evaluator has any connections 
or conflicts with the applicant which would raise questions of bias. By way of 
example, in one embodiment, reader qualification questions include whether the 

1 5 evaluator served as nominator or recommender for the applicant and whether the 
evaluator has any connection with the applicant such as having attended the same 
high school or college as the applicant or that the evaluator has any personal reason 
not to read the applicant. In this example, a yes to any reader eligibility question 
disqualifies the reader from reading the applicant. 

20 [0146] The system then checks (1006) whether the reader is qualified to read the 
applicant, and if not, the reader is disqualified (1008) from reading the applicant and 
the system again queues (1002) the applicant package with another available reader. 
[0147] After verification (1006) of reader eligibility to evaluate or read the 
applicant package, the evaluator verifies (1010) whether the applicant is eligible 

25 based on the requirements of the competition system. To facilitate this, the system 
presents the necessary information extracted from the applicant package on the 
reader's display along with a statement of applicant eligibility requirements. The 
system then determines (1012) from the reader's verification whether the applicant is 
eligible and if the applicant is not eligible, the system marks the applicant package to 

30 be in the non-select segment (112) and indexes the applicant in the non-select list. 
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[0148] If the applicant is determined (1012) to be ineligible for reading, the 
algorithm (1000) marks (1014) the applicant package to be in the non-select segment 
(112) and lists (1014) the applicant in the non-select list. In this case, the algorithm 
(1000) is thereafter finished as shown by the done indicator (1020). 
5 [0149] If the applicant is (1012) eligible, the reader then reads (1016) the 
applicant package. Reading is facilitated by the side-by-side display of the read 
variable scoring area and an information display. FIGS. 20-21 show views of an 
exemplary display for reading. In the information display, the system can display 
applicant data segments which are sections of data extracted from one or more forms 

1 0 in the applicant package or the system can display complete forms. The system also 
blocks information which is of a personal nature such as the applicant's social 
security number (SSN), email address, mailing address, alien registration number, 
date of alien registration number issuance, and family information such as gross 
annual income, family size, and number of family members currently in college. 

1 5 When the reader clicks or selects a variable, the system will pull up the relevant data 
segments or applicant package forms and displays them in the information area for 
use by the reader. 

[0150] The read variables are determined by the administrators of the 
competition system and the scores achieved by applicants on the read variables are 

20 what determines whether they are ultimately successful in the competition. By way 
of example, the read variables for one embodiment in the academic scholarship 
arena include the non-cognitive variables 1) positive self-concept, 2) realistic self- 
appraisal, 3) understanding/navigation of a social system, 4) preference of long- 
term goals over short-term goals, 5) availability of a strong support person, 6) 

25 leadership experience, community service, 7) interest/knowledge in a non-school 
field, 8) community service, as well as the cognitive-related variables 9) curriculum 
rigor, 10) academic achievement, 11) essay response. 

[0151] In one embodiment, the scoring during evaluation is by a numerical scale 
as follows: 1 indicates a negative score, 2 indicates a neutral score, 3 indicates a 
30 positive score, and 4 indicates an excellent score. In some variable cases, less than 
the total number of scores will be available. By way of example, in the case of the 
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non-cognitive variable presence of a strong support person, only the two scores of 2 
or 3 are allowed. 

[0152] After the reader has completed and submitted the reading of the applicant 
package, the system determines (1018) whether the applicant package has been read 
5 by the required number of different readers. Use of multiple readings has many 
benefits including, but not limited to, ensuring greater fairness and providing 
compensation for the event of one reader providing a randomly "unfair", inequitable, 
or inaccurate read which is not detected by the system. In one embodiment, two 
independent reads are required but more reads can be used within the scope of the 
1 0 invention. If the required number of readers has not read the applicant package, the 
system queues (1002) the applicant package with another available reader and 
continues as before. Otherwise, the reading phase for the applicant package is 
complete as shown by the done indicator (1020). 

[0153] Referring to FIG. 11, shown is a flowchart of the steps traversed in the 
15 operation of reader selection and training for participation in the algorithm (1000) of 
FIG. 10. 

[0154] First, the number of readers is determined (1 102). In one embodiment, 
this is done manually by the system administrator based on the number of candidates 
anticipated and either (a) the number of reads per reader desired or (b) the length of 

20 the reading phase considered in conjunction with the average number of reads per 
reader per day. Once the number of desired readers is determined, reader 
applications are collected (1 104) and evaluated (1 106). In one embodiment, readers 
are filtered to meet minimum criteria related to the nature of the competition to be 
conducted. By way of example, for a competition system for military officer 

25 selection, readers would likely be required to have minimum qualifications which 
include military experience or familiarity, officer selection experience, and so forth. 
The administrators of the system then select (1 106) the readers. 
[0155] The readers are then trained (1 108) with the goals and procedure of the 
competition system. Following the initial orientation, the readers are trained (1110) 

30 about the read variables and the scoring rubric. In training the readers on the scoring 
process, the readers are first trained (1 1 12) by paper and pencil scoring exercises 
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which facilitates a personal review procedure whereby trainers may personally 
discuss the scoring rubric with the reader trainee. Once the readers have 
demonstrated a basic competence during the paper exercises, the readers are then 
trained (1 1 14) by computer-based exercises which substantially mimic the actual 
reading displays and use actual prior candidate packages (without personal 
identifying information). As with the paper exercises, reader trainers are available to 
reinforce the scoring rubric by discussions of the methodology and balancing of 
factors which go into scoring the read variables. In closing the electronic reading 
exercises, an evaluation is conducted (1116) to ensure that the readers are reading at 
the level mandated by the competition system requirements. After the readers pass 
the evaluation, the reader training phase is finished as shown by the done indicator 
(1118). 

[0156] Referring to FIGS. 12a- 12b, shown is a flowchart of an exemplary 
algorithm (1200) of the steps traversed in operation of finalist selection in FIG. 4. 
[0157] Referring to FIG. 12a, shown is part 1/2 of the algorithm (1200). Finalist 
selection occurs (1202) after the reading is finished. The number of competition 
winners to be chosen is read (1204) (or alternatively input by a system operator) and 
the read variable scores for each candidate are summed (1206) to produce read 
variable sums. By way of example, in one embodiment, 1 1 read variables are scored 
each scored by two evaluators. Thus, for each candidate, there are 22 total read 
variable scores which are then summed together to provide the read variable sum for 
that candidate. After summing the read scores, software variables to be used in later 
processing are initialized (1208). The candidates are then grouped (1210) by read 
variable sums. By way of example, if 25 candidates out of a pool of 400 candidates 
achieve a read variable sum of 80, then these 25 candidates form one group and the 
other 375 candidates form one or more other groups as dictated by their read variable 
sums. 

[0158] Once the candidates are clustered into groups by read variable sums, the 
groups are ranked (1212) from highest read score sum to next lowest read score sum 
and so on until all groups are ranked. At this point, the actual selection process 
begins by first selecting (1214) the highest ranking group as the active group. Next, 
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the algorithm (1200) determines (1216) whether the total number of candidates 
already selected (zero at this point) plus the number of candidates in the active group 
exceeds the total number of competition winners to be selected. If the total number 
of candidates already selected (again, zero at this point) plus the number of 
5 candidates in the active group is less than the total number of competition winners to 
be selected, all candidates in the active group are selected (1218). Selection entails 
the candidates being designated as finalists by a entry being made in the finalist list 
and their candidate packages being flagged as a finalist. The algorithm (1200) then 
returns to block (1214) and selects (1214) the next highest ranking group as the 

1 0 active group and continues as described before except that now, the variable 

describing the total number of candidates already selected is no longer zero. The 
loop consisting of the steps described, (1214) to (1218) inclusive, continues until the 
first occurrence where the total number of candidates already selected plus the 
number of candidates in the active group is determined to exceed (1216) the total 

1 5 number of competition winners to be selected. At this point, the algorithm (1200) 

leaves the loop, steps (1214) to (1218), and continues (via the connector "C" (1220)) 
to the second level or phase 2 of selection which begins at the step (1222) shown in 
FIG. 12b. 

[0159] Referring to FIG. 12b, shown is part 2/2 of the algorithm (1200). This 
20 part of the algorithm (1200) is an exemplary embodiment for tie-breaking. Tie- 
breaking refers to the method of selecting a subset of an active group of candidates 
in order to end up with the same number of selected candidates (i.e. finalists) as the 
total number of competition winners to be selected. 

[0160] The embodiment shown in FIG. 12b uses 1 1 read variables during the tie- 
25 breaking process. However, part 2/2 of the algorithm (1200) as shown in FIG. 12b 
can be used with any number of read variables. The block (1222) is a loop control 
which limits the number of iterations to a maximum equal to the number of read 
variables available (1 1 read variables in the shown embodiment). First, the 
algorithm (1200) determines (1224) whether the number of selected candidates 
30 (finalists) exactly equals the total number of competition winners to be selected, and 
if it does, the selection process is finished as indicated by done indicator (1226). 
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Otherwise, the algorithm (1200) selects (1228) the highest priority read variable 
which was not already selected and reads (1230) the weight to be applied to that read 
variable. 

[0161] Read variable priority is a ranking or scaling of the various read variables 
5 for use during tie-breaking. The read variable priorities are, in an embodiment and 
for reasons of fairness, determined prior to the reading process. In one embodiment, 
read variable priorities are determined by partner group and not globally over the 
entire pool of candidates. In this embodiment, the partner groups are free to 
determine the read variable priorities in ways which makes the most sense relative to 
1 0 the characteristics of the candidates of that partner group. The read variable having 
the highest priority will be used first in any tie-breaking which needs to be done. 
Then, if the total number of selected candidates is not yet reached, the next highest 
variable is used and so on until the total number of allotted awards are matched to 
selected candidates. 

1 5 [0162] Once the highest priority read variable is determined, it is weighted 
(1232) by a predetermined amount. Further to the last embodiment, the 
predetermined amount, like the priority ordering of the read variables, is determined 
before the reading process. Weighting refers to multiplying the selected read 
variable by the predetermined weight for each candidate in the active group. By way 

20 of example, for a competition system applying two reads over 1 1 variables for each 
candidate, there are two read variables for each candidate which are weighted and 
the predetermined weight to be applied during first round tie-breaking is 2. After 
weighting the read variables, a new read variable sum is computed (1234). Next, the 
candidates (of only the active group) are first clustered (1236) into subgroups or 

25 cells such that (a) each subgroup has candidates all having the same new read 
variable sum (which incorporates the weighted variables) and (b) all candidates 
having the same new read variable sum are in the same subgroup. Next, the 
subgroups are ranked (1238) by from highest weighted read variable sum 
progressively down to the lowest weighted read variable sum. 

30 [0163] On the first pass through the tie-breaking loop (block (1222) to block 
(1248) inclusive), only one read variable weight is applied. On each successive 
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iteration through the tie-breaking loop another read variable is weighted. Read score 
sums in a given iteration through the tie-breaking loop incorporate the weighted read 
scores of all previous iterations (e.g. once a read variable is weighted, the weighted 
variable is used for the rest of the tie-breaking rather than reverting to the original 
5 non- weighted score). 

[0164] The tie-breaking selection process now continues similarly to the 
selection process as described in reference to FIG. 12a. The highest subgroup is 
designated (1240) as the active subgroup. Next, the algorithm (1200) determines 
(1242) whether the total number of selected candidates plus the candidates in the 

1 0 active subgroup exceeds the total number of competition winners to be selected. If 
not, the candidates in the active subgroup are selected (1244) and the algorithm 
(1200) continues back and selects (1240) the next highest ranked subgroup as the 
active subgroup and continues as before. The act of selecting of a candidate means 
that the candidate has been converted to finalist status. At this point, all that is done 

15 is that the candidate is indexed into a selected list. The loop consisting of block 

(1240) to block (1244) is iterated until the total number of selected candidates plus 
the candidates in the active subgroup is determined to exceed (1242) the total 
number of competition winners to be selected. When this occurs, the highest 
ranking subgroup having candidates who have not been selected is designated (1246) 

20 the active group. Next, the algorithm (1200) checks (1248) whether the total 
number of competition winners to be selected equals the number of selected 
candidates and if it has, algorithm (1200) has finished the selection process (as 
shown by block 950). Otherwise, the algorithm (1200) continues back and 
determines (1222) the next highest priority read variable and continues as before. 

25 [0165] Generally, this embodiment of tie-breaking is very effective in narrowing 
out of a group of candidates the necessary subgroup to fill out the total number of 
candidates to be selected. The only complication is when two or more candidates 
receive exactly the same score on each read variable and also just happen to fall 
within the read score hierarchy such that only a subset of them can be chosen to fill 

30 out the allotment of awards. In this case, successive weighting and ranking will not 
be able to differentiate between them so as to fill out the award allotments. This 
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situation can be handled by having backup variables which normally are not used for 
scoring and ranking which can be used. These backup variables could also be scored 
by the readers, or alternatively, they could be taken from candidate information such 
as GPA, normalized GPA, etc. 
5 [0166] Once all of the finalists have been selected, the system implements the 
actual conversion of the candidates to finalists by designating all candidates as a 
finalists in the their respective candidate packages, adding the candidates to the 
finalist list, and deleting the candidates from the nominee list. 
[0167] It is noted that in one embodiment, candidate finalists are subject to 

1 0 confirmation and if confirmed, are required to accept prior to being confirmed as a 
competition winner. In this embodiment, some candidate finalists will not be 
confirmed and others who are confirmed may not accept. Thus, in order for the 
predetermined number of competition winners to be met, the system must select new 
competition finalists for each competition finalist who is ultimately not confirmed or 

1 5 who does not accept even when confirmed. In one embodiment, the system selects 
additional finalists by continuing from where the phase 1 and phase 2 selection 
systems left off. The selection of additional finalists can be done periodically, such 
as once a week, or can be done more frequently such as once a day or even each time 
notification of non-acceptance or non-confirmance is received. 

20 [0168] Referring to FIGS. 12c-12e, shown is an example of an application of tie- 
breaking. Referring to FIG. 12c, shown is an example of the priority ordering of 
evaluation variables for two partner groups in one embodiment. 
[0169] As an example of tie-breaking, one embodiment concerns the selection of 
individuals for employment. In this embodiment, the variables over which the 

25 candidate packages are evaluated are curriculum rigor, overall academic 

achievement, positive self-esteem, realistic self appraisal, understanding and 
navigation of social and organizational systems, preference for long-term goals over 
immediate gratification, leadership experience, community service, self-motivated 
acquisition of knowledge or skills, persuasiveness, language structure and 

30 expression ability. These variables are chosen to maximize the predictive ability of 
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the competition system and it is noted that in competitions employed for other 
purposes or with different goals, these variables may be chosen to differently. 
[0170] In this embodiment, applicants are divided into partner groups which 
include partner groups for engineering and business management. Within the 
5 partner group for engineering the evaluation variables are ordered for priority as: 

overall academic achievement, curriculum rigor, language structure and expression 
ability, preference for long-term goals over immediate gratification, self-motivated 
acquisition of knowledge or skills, understanding and navigation of social and 
organizational systems, leadership experience, persuasiveness, positive self-esteem, 

1 0 realistic self appraisal, and community service. In comparison, within the partner 
group for business management the evaluation variables are ordered for priority as: 
understanding and navigation of social and organizational systems, leadership 
experience, positive self-esteem, language structure and expression ability, 
persuasiveness, preference for long-term goals over immediate gratification, self- 

1 5 motivated acquisition of knowledge or skills, overall academic achievement, 

curriculum rigor, realistic self appraisal, and community service. It is noted these 
orderings per partner group reflect the goals of the particular competition and would 
likely be chosen differently for different competition objectives. 
[0171] Referring to FIG. 12d, shown are an example of the evaluation results for 

20 two candidates in an embodiment. 

[0172] As an example, two candidates are in competition for an engineering 
position. As part of the qualification and eligibility process, each was determined to 
have the necessary engineering degrees and so on. During the reading process, 
candidate # 1 received for reader #1 : 4, 4, 2, 3, 3, 2, 4, 2, 3 and, for reader #2: 3, 4, 2, 

25 2, 4, 2, 3, 4, 4, 3, 2. Candidate #2 received for reader #1 : 4, 4, 3,3,3, 2, 4, 4, 3, 2, 2 
and for reader #2: 2, 4, 4, 4, 3, 2, 4, 2, 2, 3, 2. 

[0173] Referring to FIG. 12e, shown is an example of tie-breaking between the 
two candidates of FIG. 12d according to the evaluation variable priorities of FIG. 
12c. 

30 [0174] Thus, in phase 1 of the selection process, both candidates have an overall 
score of 66 and so a decision of which to choose cannot be made. In a situation 
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where only one of these candidates is to be selected, resort is made to the phase 2 or 
tie-breaking selection process which uses evaluation variable priorities. For the 
engineering partner group, the first priority variable is overall academic achievement 
(the second variable in the global listing of variables). In the example, both 
5 candidates have the values of 4, 4 in their two readings. If the weighting is 2, for 
example, these scores become 8 and 8 for both candidates and their overall scores 
become 74 for both. The second priority variable for the engineering groups is 
curriculum rigor (the first variable in the global evaluation variable listing). 
Candidate #1 received scores of 3, 3 for this variable whereas candidate #2 received 

10 4, 2. Applying a weighting of 2 again, these scores become 6, 6 and 8, 4, 

respectively and both candidates receive an overall score of 82. The third priority 
variable for the engineering partner group is language structure and expression 
ability (the last variable in the global evaluation variable order) for which candidate 
#1 received 3, 2 and candidate #2 received 2, 2. Thus, applying a weighting of 2 for 

15 this variable results in scores of 6, 4 and 4, 4, respectively. Thus, candidate #1 now 
has a total score of 87 and candidate #2 now has a score of 86 and the tie has been 
broken in favor of candidate #1 . 

[0175] Referring to FIG. 13 shows a flowchart of an exemplary algorithm (1300) 
of the steps traversed in operation of competition finalist confirmation shown in 
20 FIG. 4. 

[0176] competition finalist confirmation refers in part to the verification of 
various data presented on candidate package forms and in part to verify the 
eligibility of the candidate relating to any new information such as may now be 
available concerning the tuition needs of the candidate finalist. In one embodiment, 

25 once candidates have passed the reading and selection phases, they are transferred to 
a scholar tracking module. Entry into the scholar tracking module allows for 
individual access by administrators as various confirmation documents arrive and 
are incorporated into the system and also allows manual update of information and 
candidate status. The scholar tracking system allows for the administrator to search 

30 by social security number (SSN), partial SSN, specific surname, and generic 

surname. The term generic surname refers to the entry of an initial set of letters 
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followed by a wildcard. The result of a generic surname search is a list of all names 
matching the query. By way of example, the query "Mus*" would result in a list 
containing Musam, Muse, and Musni provided those names were in the database 
queried. 

[0177] The algorithm (1300) causes (1302) correspondence to be sent to the 
candidate finalists which announces their finalist status and which requests all 
necessary confirmation documents. In one embodiment, the required confirmation 
documents include a complete high school transcript (covering at least the last 3 
years attended) or a general equivalency degree (GED) as appropriate, any tribal 
documents if the finalist is Native American (note that some Native American tribes 
have only one document to indicate tribal membership whereas others may have 
multiple documents which together indicate tribal membership), a United Negro 
College Fund (UNCF) Scholars information form (discussed in detail in the next 
paragraph), a copy of the finalist's college/university admissions letter indicating 
admittance, a letter/document for each scholarship (outside of this competition) 
which the finalist has been awarded indicating the amount of award, and any 
documents for each loan the finalist has indicating the amount and terms of the loan. 
In a variation of this embodiment, confirmation documents additionally include 
documents attesting to citizenship or primary residency, affidavits and documents 
regarding community service, documents regarding employment, documents 
regarding honors and awards, and affidavits and documents regarding leadership 
experience. 

[0178] The UNCF Scholars information form is a form developed to standardize 
institution information disclosure relating to financial grants to students. It has been 
determined through actual experience that the levels of detail provided by colleges 
and universities in response to inquiries concerning financial grants to students can 
vary wildly from institution to institution. For example, some universities and 
colleges do not report 3 rd party scholarships in student financial reports. In response 
to this, a standardized form was developed which specifies what information needs 
to be disclosed. In one embodiment, the UNCF Scholars information form requires 
disclosure of all scholarships, grants, loans, and other financial assistance to the 
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student, both from the academic institution as well as 3 rd parties. This level of detail 
is needed, in this embodiment, as the goal of the competition system in which it is 
used is to (1) complete the financial needs of competition winners by providing any 
required academic tuition not met by financial assistance already in place and 
5 additionally to repay in full all loans (so the student will not have that obligation 
later). 

[0179] Once confirmation documents are received, they are incorporated into the 
system by the document receiving subsystem embodiments of which are similar to 
the procedure described in reference to FIGS. 4 and 5 except that documents are 
1 0 stored (1 304) in the candidate database and linked immediately with the rest of the 
corresponding candidate package. 

[0180] In the actual confirmation process, a candidate package is selected 
(1306). The algorithm (1300) then determines (1308) whether all required 
documents are present. If all the required confirmation documents are present, the 

1 5 candidate confirmation documents are evaluated (1310). In one embodiment, the 
evaluation of candidate confirmation data is done by a system administrator. The 
algorithm (1300) then determines (1312) whether the candidate has been confirmed, 
and if so, the candidate is converted (1 3 14) to scholar status. Conversion to 
competition winner status includes the marking of the candidate package as being in 

20 the competition- winner segment (116) and the addition of the candidate to a 
competition winner list. The competition winner list is the master list of all 
competition winners for use by the system. From this point on, the candidate is 
referred to as a competition winner and his or her package is referred to as a 
competition winner package. 

25 [0181] After conversion ( 1 3 1 4) of a candidate to a competition winner, the 

algorithm (1300) determines (1316) whether any more candidate confirmations are 
outstanding, and if so, the algorithm (1300) selects (1308) another candidate package 
and continues as before. If the algorithm (1300) determines (1316) that no candidate 
packages are outstanding, the confirmation process is finished as shown by the done 

30 indicator (1316). 
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[0182] If the algorithm (1 300) determines (1 308) that one or more required 
confirmation documents are missing, the candidate is converted (1320) to a non- 
select by marking the candidate package as being non- select and listing the candidate 
in the non-select list. Thereafter, the algorithm (1300) determines (1316) whether 
5 any candidate packages still need to be confirmed, and continues as described 
previously. 

[0183] If the algorithm (1300) determines (1312) that the candidate is not 
confirmed, the candidate is converted (1320) to a non-select as described previously 
and the algorithm (1300) thereafter determines (1316) whether any candidate 
10 packages still need to be confirmed, and continues as described previously. 

[0184] Referring to FIG. 14 shows a flowchart of an exemplary algorithm (1400) 
of the steps traversed in operation of competition winner tracking and payment and 
tracking in FIG. 3 . 

[0185] While the algorithm (1400) is optimized for the administration of 
1 5 academic scholarships, it is noted that with minor adaptations, this algorithm can 
monitor the continued performance of military or civil officers, employee 
performance, and so on. 

[0186] Competition tracking refers to the monitoring of scholar progress, 
updating of competition winner files to add academic performance and other 

20 information, and re-qualification that the competition winner meets the requirements 
of the competition system. As such, the competition winner tracking system must 
periodically update the competition winner files. To do this, the algorithm (1400), 
after determining that it is time to re-qualify one or more competition winners (such 
as just before an academic year in an academic competition system), sends (1402) 

25 correspondence requesting any needed information. In one embodiment relating to 
academic scholarships, academic transcripts are requested once a year at the 
completion of the year of study. Further to this embodiment, prior to each further 
scholarship payment, the scholar is asked whether he or she wishes to continue in the 
scholarship. 

30 [0187] In one embodiment, the system maintains a scholar tracking action list 

for scholars in order to keep track of when re-qualification is necessary. The system 
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determines, by accessing the scholar tracking action list, when an action, such as 
requesting re-confirmation documents is required. In another embodiment, this 
information is kept separately for each scholar. 

[0188] All re-qualification documents received in support of the scholar are 
5 stored (1 404) into the competition- winner segment (116) and associated with the 
competition winner package. As with initial confirmation documents, documents 
requested in response to updating a competition winner's package can be received 
electronically or by hardcopy and are incorporated into the system by the same 
system and process as described in reference to FIGS. 4 and 5, with the difference 
1 0 that the documents are stored in the scholar database and associated with the scholar 
package. 

[0189] Next, a first competition winner package is selected (1406). Next, the 
algorithm (1400) determines (1408) whether the re-qualification documents are all 
present and if so, the algorithm (1400) allows evaluation (1410) of the re- 

1 5 qualification materials to determine if the scholar re-qualifies. Both the checks if all 
required documents are present and the evaluation of the competition winner's re- 
qualification can be implemented by automation, by human reviewers, or a 
combination of both. In one embodiment relating to the field of academic 
scholarship competitions, both the checking if all required documents are present 

20 and the evaluation of the scholar's re-qualification are done by human reviewers. 
Further to this embodiment, the algorithm (1400) allows the human reviewers to 
quickly search in scholar packages for specific documents by keyword, document 
type, document receipt date, document originator, and so forth. Further, the 
algorithm (1400) allows for simultaneous display of portions of, or the complete 

25 contents of, any re-qualification materials (or indeed any form in the scholar 
package) with the re-qualification rubric and/or decision input form. 
[0190] After evaluation (1410), the algorithm (1400) determines (1412) whether 
the competition winner has re -qualified and if so, the new amount of the scholarship 
award is calculated (1414). After the scholarship award amount has been 

30 determined (1414), a check is printed and sent (1416) to the competition winner's 
educational facility. 
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[0191] Next, the algorithm (1400) determines (1418) if all competition winners 
have been passed through the re-qualification process, and if so, the re-qualification 
process is finished (as shown by done indicator (1420). Otherwise, the algorithm 
(1400) selects (1406) another scholar package which has not been re-qualified and 
continues as described before. 

[0192] If the algorithm (1400) determines (1408) that not all required re- 
qualification documents are present, the algorithm (1400) alerts and facilitates 
(1422) competition winner reviewers to follow up with the competition winner. If 
the re-qualification materials are competed (1424), the algorithm (1400) evaluates 
(1410) the scholar's re-qualification materials and continues as discussed previously. 
If the algorithm (1400) determines (1424) that the re-qualification materials is not 
complete, the algorithm (1400) determines (1426) whether the competition winner 
has at least one semester of sabbatical left under the terms of the scholarship. If the 
competition winner does have at least one semester of sabbatical left, the 
competition winner is designated (1428) as being on sabbatical and the competition 
winner's sabbatical allotment is decremented (1428) by one (1). Thereafter, the 
algorithm (1400) determines (1418) if all competition winner packages have been 
reviewed and continues as previously discussed. If the competition winner is 
determined (1426) to have no sabbaticals left, the competition winner's scholarship 
is terminated (1430) and the competition winner's package is updated (1430) to 
reflect this status. After terminating (1430) the scholar's scholarship, the algorithm 
(1400) determines (1418) whether any scholars have not been processed through the 
re-qualification process and continues as previously described. 
[0193] If the competition winner is determined (1 4 1 2) to not be re-qualified, the 
algorithm (1400) determines (1426) whether the competition winner has any 
sabbaticals left and continues as previously discussed. 

[0194] Referring to FIG. 14a, shown is a flowchart of the steps traversed in an 
exemplary method (1450) for the tracking of competition winners and non-selected 
applicants. 

[0195] In one embodiment for the distribution of academic scholarships, the 
tracking and re-qualification subsystem (260) implements tracking and information 
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accumulation on both scholars and non-selected applicants. The algorithm (1450) 
operates by the administration of surveys and other information retrieval activities 
based on the time of year. The algorithm (1450) first checks (1452) if it is the start 
of the academic year, and it it is, the algorithm (1450) updates scholar data through 
5 request (1454) of the re-qualification materials as discussed in reference to FIG. 14. 
As discussed in reference to FIG. 14, once the re-qualification materials are 
received, they are incorporated (1456) into the system. If it is not (1452) the start of 
the academic year, the algorithm (1450) checks (1458) if it is the time for the 
orientation of new scholars and if it is, new scholar surveys are administered (1460) 

10 to the new scholars. Thereafter, this information is incorporated (1456) into the 

system. If it is not (1458) orientation time, the algorithm (1450) checks (1462) if it 
is time for the bi-annual surveys and if it is, career development surveys (1464) and 
academic support surveys (1466) are administered to the active scholars. If it is not 
(1462) time for the bi-annual active scholar surveys, the algorithm (1450) checks 

15 (1468) if it is graduation time for any active scholars and if it is, graduate surveys are 
administered (1470) to the graduating scholars. Thereafter, this information is 
incorporated (1456) into the system. If it is not (1470) a graduation time, the 
algorithm (1450) waits again until the first of the academic year (1452) and 
continues as previously described. 

20 [0196] Additionally, information supplied by active scholars and scholar alumni 
may be received (1472) at any time. This can occur by mailed in material 
submissions, facsimile transmissions, website submissions, and so forth. As with 
the previously discussed survey information, this information is incorporated (1456) 
into the system. 

25 [0197] Information is also received (1474) from third parties regarding non- 
selected applicants. This received information is very important for verifying and 
improving the selection rubric and criteria employed by the system. Non-selected 
applicant data is extracted from third parties which get this information during 
transactions with the non-selected applicants during situations such as, but not 

30 limited to, applications to other scholarships, credit cards, colleges or universities, 
etc. as well as from targeted promotions designed to elicit tracking information. 
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[0198] Improvement of the system rubric and evaluation criteria can be 
effectuated by tracking the performance of both competition winners and non- 
selected applicants and comparing the later performance of both types of individuals 
with their predicted performance as derived from the system rubric through the 
5 evaluation variables. To do this, each competition winner' s tracked performance is 
evaluated and scored over one or more performance variables. In one embodiment, 
these variables include annual salary, gaps between employment periods, number of 
people managed, number of positive appearances on the news or on television, job 
satisfaction, community leadership, positive contributions to society, and so forth. 

1 0 Analysis is conducted of these performance variable scores in reference to the 

individual's original evaluation variable scores to determine the accuracy of each 
evaluation variable in predicting the eventual resultant performance of the 
individual. This analysis is conducted for all competition winners and non-selected 
applicants. The accuracy of predicting eventual resultant performance of each 

1 5 performance variable can thus be determined for each evaluation variable. 
Additionally, this analysis can be conducted separately for any subgroup of 
competition winners and for a corresponding subgroup of non-selected applicants to 
determine the significance of each evaluation variable in correctly differentiating 
individuals who went on to perform significantly from those who did not. 

20 [0199] Referring to FIG. 15, shown is an exemplary display presented to an 
evaluator. An evaluator is given the opportunity to either get an applicant for 
evaluation or to quit. 

[0200] Referring to FIGS. 16-17, shown are two views of an exemplary display 
for use during determination of evaluator eligibility. Shown are selected information 

25 from an applicant package. In this example, shown are the candidate's gender, 
birthdate, dependency status, race/ethnicity, languages spoken, summary of high 
school attendance, general equivalency degree (GED) status, and future goals. 
Additionally shown is evaluator eligibility question guidelines and three evaluator 
eligibility questions for the evaluator to answer. Note that in other embodiments, 

30 different questions and different numbers of total questions are used. Determination 
of evaluator eligibility and subsequent redirection of applicant packages to another 
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evaluator when the evaluator is determined ineligible to evaluate the applicant 
package helps ensure an unbiased evaluation. Additionally, evaluator eligibility is 
documented in the system for later use, if needed, to counter any received allegations 
of impropriety. 

5 [0201] Referring to FIGS. 18-19, shown are two views of an exemplary display 
for use during determination of candidate eligibility. Shown is information extracted 
from a candidate package and four questions regarding eligibility of the candidate 
package to be evaluated. In this example, the questions regarding the candidate 
package eligibility are whether the candidate is eligible based on high school 
1 0 graduation or GED completion, high school grade point average (GP A), not having 
attended college or university, and sufficient information for evaluation. Note that 
in other embodiments, different questions and different numbers of total questions 
are used. 

[0202] Referring to FIGS. 20-21 , shown are exemplary displays for use during 

1 5 evaluation of candidate packages. Shown in FIG. 20 is the typical split-screen 

display used in one embodiment. Shown are two windows or areas. One area (the 
data area) contains information from the nominator's form for the candidate. The 
other area (the control and response area) contains various controls or hyperlinks 
which control the information displayed in the first area. In the shown embodiment, 

20 the controls allow the evaluator to control the data area to show applicant form data, 
nominator form data, or recommender form data. By selecting, such as by use of a 
mouse, an evaluator directs the system to display information from the selected 
record in the data area. The control and response area also contains 1 1 evaluation 
variables requiring evaluator responses. In this embodiment, shown are the variables 

25 1) curriculum rigor, 2) overall academic achievement, 3) positive self-concept, 4) 
realistic self-appraisal, 5) understanding/navigation of a social system, 6) 
preference of long-term goals over short-term goals, 7) leadership experience, 
community service, 9) self-directed acquisition of knowledge in a non-school field, 
8) community service, 10) availability of a strong support person, 11) structure and 

30 use of language in essays. By selecting a particular one of the evaluation variables, 
the evaluator directs the data area to scroll to or extract information from the 
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displayed record which is relevant to the selected variable. For example, as shown 
the data area of FIG. 20 shows information relevant to curriculum rigor. The data 
area shows this information whenever the evaluator indicates, by selecting the 
curriculum rigor variable, that the evaluator wishes to evaluate the curriculum rigor 
variable. 

[0203] FIG. 20a shows the screen shot of FIG. 20 with additional control 
windows. The window below and to the left of the two evaluation windows as 
shown in FIG. 20 is a control window allowing the evaluator to access the stored 
image of a document from the applicant package under evaluation. This allows the 
evaluator to view the original document image which is displayed in the data area. 
This capability is also used by authorized personnel when deciding situations of 
possible fraud. The window shown to the right of the two evaluation windows as 
shown in FIG. 20 is a control window allowing the evaluator to find a document 
view. As shown, a document can be found by searching such as by social security 
number and form type or content. 

[0204] Referring to FIGS. 22-25, shown are exemplary displays of candidate 
progress detail reports. Shown in FIG. 22 is an applicant package progress report 
detailing applicant packages grouped by complete/incomplete and sorted by social 
security number (SSN). Shown in FIG. 23 is an applicant package progress report 
grouped by complete/incomplete and sorted by applicant name. Shown in FIG. 24 is 
an applicant package progress report grouped by complete/incomplete, by ethnicity, 
and sorted by social security number (SSN). Shown in FIG. 25 is an applicant 
package progress report grouped by complete/incomplete, by ethnicity, and sorted by 
applicant name. In one embodiment, access to these reports is restricted such that 
only system administrators are able to bring up such reports. 
[0205] Referring to FIGS. 26-28, shown are exemplary displays of completed 
applicant package reports. FIG. 26 shows a report listing all completed applicant 
packages sorted by applicant social security number. FIG. 27 shows a report listing 
all completed applicant packages sorted by applicant name. FIG. 28 shows a report 
listing all completed applicant packages sorted by submit date of the applicant 
(nominee) form. 
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[0206] Referring to FIGS. 29-30, shown are exemplary displays of possible 
duplicate document reports. FIG. 29 shows a listing of possible duplicate forms 
listed by applicant social security number. FIG. 30 shows a report of possible 
duplicate forms listed by applicant name. 

[0207] Referring to FIGS. 31-34, shown are exemplary display of candidate 
detail listing reports. Shown are candidate detail data. FIG. 3 1 shows applicant 
detail data sorted by social security number. FIG. 32 shows candidate detail data 
sorted by applicant name. FIG. 33 shows applicant detail data ordered by one 
partner group sorted by social security number. FIG. 34 shows candidate detail data 
ordered by partner group and secondly by applicant name. 

[0208] Referring to FIG. 35, shown is an exemplary report of a nominator floater 
with possible applicant or key form (nominee form) ordered by applicant last name 
and secondly by social security number. This report lists all nominator floaters and 
likely corresponding key forms (nominee forms), if the system discovers any. 
[0209] Referring to FIGS. 36-37, shown are exemplary display of nominator 
floater detail reports. FIG. 36 shows a report listing nominator form floaters ordered 
by applicant social security number. FIG. 37 shows a report listing nominator form 
floaters ordered by applicant last name and secondly by social security number. 
[0210] Referring to FIGS. 38-40, shown are exemplary display of nominator 
detail reports. These reports list nominator detail data. FIG. 38 shows nominator 
detail data ordered by applicant social security number. FIG. 39 shows nominator 
detail data ordered by applicant name. FIG. 40 shows nominator detail data ordered 
by nominator name. 

[02 1 1 ] Referring to FIG. 4 1 , shown is an exemplary report of recommender 
floaters with possible key form (nominee form) linkages ordered by applicant last 
name and secondly by social security number. This report lists all recommender 
floaters and likely corresponding key forms (applicant forms), if the system 
discovers any. 

[0212] Referring to FIGS. 42-43, shown are exemplary display of recommender 
floater detail reports. FIG. 42 shows a report listing recommender form floaters 
ordered by applicant social security number. FIG. 43 shows a report listing 
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recommender form floaters ordered by applicant last name and secondly by social 
security number. 

[0213] Referring to FIGS. 44-46, shown are exemplary display of recommender 
detail reports. These reports list recommender detail data. FIG. 44 shows 
5 recommender detail data ordered by applicant social security number. FIG. 45 
shows recommender detail data ordered by applicant name. FIG. 46 shows 
recommender detail data ordered by recommender name. 

[0214] While the invention herein disclosed has been described by means of 
specific embodiments and applications thereof, numerous modifications and 
1 0 variations could be made thereto by those skilled in the art without departing from 
the scope of the invention set forth in the claims. 
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